{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](https://cdn-images-1.medium.com/max/800/0*yoBvg_ik5WZTQkLq.)\nImage source *https://cdn-images-1.medium.com/max/800/0yoBvg_ik5WZTQkLq.*","metadata":{"_uuid":"e3576c35343ac87efda3d7f0dcdf307aed167239"}},{"cell_type":"markdown","source":"Machine Learning Methods have for long been infamous for being BlackBox or \"un-interpretable\". The growing popularity and complexity of the models worsens the case. There is a trade-off between accuracy and interpretability.\n\n![](https://image.slidesharecdn.com/kasiainterpretablemachinelearning-171219021018/95/interpretable-machine-learning-using-lime-framework-kasia-kulma-phd-data-scientist-aviva-16-638.jpg?cb=1513658331)\n\nThere are some domains especially in the world of finance and healthcare where data scientists often end up having to use more traditional machine learning models (linear or tree-based). The reason being that model interpretability is very important for the business to explain each and every decision being taken by the model. However, this often leads to a sacrifice in performance. This is where complex models like ensembles and neural networks typically give us better and more accurate performance (since true relationships are rarely linear in nature). We, however, end up being unable to have proper interpretations for model decisions.I try to address this gap by using **model-agnostic methods**, which is independent of the model being used.\n\nThe question you might find yourself asking at this point of time is : ***\"Why do I need an interpretable model, I don't work in finance?\"***\n\n> **Human curiosity and Learning**- When you will show your magic-like model to someone, the first thing that person will ask is ,*How did it do this?* What are you gonna say , *I don't know, I just fed the input  to the model, tuned some hyperparameters and got this output*. OF COURSE NOT!\n\n> **Building Trust**- When you are selling your ML product to a prospective buyer , why should he trust your model? How can he know the model will produce good results under all circumstances? Interpretability is required to increase social acceptance of ML Models in our day-to-day lives.\n\n> **Debugging**- When you are trying to reason an unexpected result or finding a bug in your model, Interpretability becomes very useful.\n","metadata":{"_uuid":"29048f4231b2cc19fdb113b2f09b128f897a2edb"}},{"cell_type":"markdown","source":"If you are not convinced yet , you can read a [story](https://christophm.github.io/interpretable-ml-book/storytime.html) to convince yourself.","metadata":{"_uuid":"012c1e6dcca4a57ad53f1218710c44f6285179fd"}},{"cell_type":"markdown","source":"Interpretability is basically of 2 types:-\n\n* **Model Specific**-  Model-specific interpretation tools are very specific to intrinsic model interpretation methods which depend purely on the capabilities and features on a per-model basis. This can be coefficients, p-values, AIC scores pertaining to a regression model, rules from a decision tree and so on. The interpretation of regression weights in a linear model is a model-specific interpretation, since – by definition – the interpretation of intrinsically interpretable models is always model-specific. \n\n* **Model-Agnostic**- Model-agnostic tools can be used on any machine learning model and are applied after the model has been trained (post hoc). These agnostic methods usually work by analyzing feature input and output pairs. By definition, these methods cannot have access to model internals such as weights or structural information.","metadata":{"_uuid":"f93480ef68304e34d5a7457f83a2fb470ccd488a"}},{"cell_type":"markdown","source":"In this kernel , we will focus on model-agnostic methods.","metadata":{"_uuid":"c660006fe8159b3867b4c1bffb64dfa8d3a68be7"}},{"cell_type":"markdown","source":"I'm using the Pima Indians Diabetes dataset for  classifying whether a person has diabetes or not based on some features.\n","metadata":{"_uuid":"3aebcd20d4627e96090585225f75c3f6de16855a"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split,GridSearchCV\nimport zipfile\nprint(os.listdir(\"../input/jigsaw-toxic-comment-classification-challenge/\"))","metadata":{"_uuid":"47890a8c7d2cb2fd80be20e929961aa4e2dda8bb","execution":{"iopub.status.busy":"2022-02-11T04:18:16.648686Z","iopub.execute_input":"2022-02-11T04:18:16.649289Z","iopub.status.idle":"2022-02-11T04:18:17.777049Z","shell.execute_reply.started":"2022-02-11T04:18:16.649218Z","shell.execute_reply":"2022-02-11T04:18:17.776051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set(style=\"white\", palette=\"colorblind\", font_scale=1.2, \n        rc={\"figure.figsize\":(12,9)})\nRANDOM_STATE = 420 \nN_JOBS=8","metadata":{"_uuid":"27a5a29b269193aa7b214a97d3d3c8041170a41d","execution":{"iopub.status.busy":"2022-02-11T04:18:17.780492Z","iopub.execute_input":"2022-02-11T04:18:17.780789Z","iopub.status.idle":"2022-02-11T04:18:17.788155Z","shell.execute_reply.started":"2022-02-11T04:18:17.780733Z","shell.execute_reply":"2022-02-11T04:18:17.786788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"../input/pima-indians-diabetes-database/diabetes.csv\")\ndf.head()","metadata":{"_uuid":"79506510a9c5c6d7e9bfd11a491dcc376340e809","execution":{"iopub.status.busy":"2022-02-11T04:19:11.673691Z","iopub.execute_input":"2022-02-11T04:19:11.674063Z","iopub.status.idle":"2022-02-11T04:19:11.731619Z","shell.execute_reply.started":"2022-02-11T04:19:11.673987Z","shell.execute_reply":"2022-02-11T04:19:11.730696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's know our features:\n * *Pregnancies*- Number of past pregnancies of the patient\n * *Glucose* -  Plasma glucose concentration(mg/dL)\n * *Blood Pressure*- Diastolic blood pressure (mm Hg\n * *Skin Thickness*- Triceps skin fold thickness (mm)\n * *Insulin*- 2-Hour serum insulin (mu U/ml)\n * *BMI*- Body mass index (weight in kg/(height in m)^2)\n * *Diabetes Pedigree Function*-  It determines whether a trait has a dominant or recessive pattern of inheritance.It is calculated when a patient has a diabetes history in the family.\n * *Age*- Age (years)\n * *Outcome*- Whether a person has diabetes or not( 0=No, 1=Yes)\n \n Insulin helps a person to use glucose in the blood to release energy. In a diabetic person , there isn't enough insulin to do that and most of the glucose remains in the blood.\n ","metadata":{"_uuid":"4a83f84ca8eda15e680b2c337aebf1af0b226d7d"}},{"cell_type":"code","source":"features=[\"Pregnancies\",\"Glucose\",\"BloodPressure\",\"SkinThickness\",\"Insulin\",\"BMI\",\"DiabetesPedigreeFunction\",\"Age\"]","metadata":{"_uuid":"f04fc4152655287ca45acbd2226bb4e080f6236e","execution":{"iopub.status.busy":"2022-02-11T04:19:13.291642Z","iopub.execute_input":"2022-02-11T04:19:13.292288Z","iopub.status.idle":"2022-02-11T04:19:13.297219Z","shell.execute_reply.started":"2022-02-11T04:19:13.292217Z","shell.execute_reply":"2022-02-11T04:19:13.296154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classes=['non-diabetic','diabetic']","metadata":{"_uuid":"e2e39dc08548cd06e9a1876a233c6f9f8f5050b8","execution":{"iopub.status.busy":"2022-02-11T04:19:13.714399Z","iopub.execute_input":"2022-02-11T04:19:13.715081Z","iopub.status.idle":"2022-02-11T04:19:13.719776Z","shell.execute_reply.started":"2022-02-11T04:19:13.714669Z","shell.execute_reply":"2022-02-11T04:19:13.718878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target=df['Outcome']\ndf=df.drop(labels=['Outcome'],axis=1)\nX_train, X_test, y_train, y_test = train_test_split(df, target, test_size=0.2, random_state=42)","metadata":{"_uuid":"52d9188bee09528cea678c55ff34f39268b153b1","execution":{"iopub.status.busy":"2022-02-11T04:19:14.116802Z","iopub.execute_input":"2022-02-11T04:19:14.117459Z","iopub.status.idle":"2022-02-11T04:19:14.129629Z","shell.execute_reply.started":"2022-02-11T04:19:14.117393Z","shell.execute_reply":"2022-02-11T04:19:14.128549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that training is not our main motive, our motive is to interpret the result.","metadata":{"_uuid":"ac6bcc7d2fd117684a9de8d2dfb3e8f00081ad79"}},{"cell_type":"code","source":"rfc=RandomForestClassifier(random_state=1234)\nrfc.fit(X_train,y_train)\n","metadata":{"_uuid":"ba7731734b4946d9ec6baca30783514c0e800e1f","_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-11T04:19:14.962975Z","iopub.execute_input":"2022-02-11T04:19:14.963640Z","iopub.status.idle":"2022-02-11T04:19:14.988613Z","shell.execute_reply.started":"2022-02-11T04:19:14.963586Z","shell.execute_reply":"2022-02-11T04:19:14.987961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How the model does for our test data?","metadata":{"_uuid":"4c3bf8a2a54c5f1980dbabdc790307278d13ff39"}},{"cell_type":"code","source":"rfc.score(X_test,y_test)","metadata":{"_uuid":"383060b7752cd9936df0e02ada244c96a9ba61e3","execution":{"iopub.status.busy":"2022-02-11T04:19:16.799362Z","iopub.execute_input":"2022-02-11T04:19:16.800038Z","iopub.status.idle":"2022-02-11T04:19:16.808831Z","shell.execute_reply.started":"2022-02-11T04:19:16.799950Z","shell.execute_reply":"2022-02-11T04:19:16.807857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not bad! \n\nLet's do some hyperparameter tuning .","metadata":{"_uuid":"d624728707384075f54e04b0a6fff73da1223f71"}},{"cell_type":"code","source":"parameters={\"n_estimators\":[10,20,50,100,200],\n           \"max_depth\":[2,3],\n           \"min_samples_split\":[2,3,4],\n           \"max_features\":('auto','log2'),\n           \"criterion\":('gini','entropy')}\nclf=GridSearchCV(rfc, parameters, cv=5)\nclf.fit(X_train,y_train)","metadata":{"_uuid":"580cc8e321f0109a3562cbac0f08aa5d98a178a6","_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-11T04:19:18.178325Z","iopub.execute_input":"2022-02-11T04:19:18.178623Z","iopub.status.idle":"2022-02-11T04:20:08.186074Z","shell.execute_reply.started":"2022-02-11T04:19:18.178578Z","shell.execute_reply":"2022-02-11T04:20:08.185243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clf.best_params_","metadata":{"_uuid":"485f22dcdc7575eb502c27058dcc3eabe6d77149","execution":{"iopub.status.busy":"2022-02-11T04:20:08.187381Z","iopub.execute_input":"2022-02-11T04:20:08.187827Z","iopub.status.idle":"2022-02-11T04:20:08.192788Z","shell.execute_reply.started":"2022-02-11T04:20:08.187574Z","shell.execute_reply":"2022-02-11T04:20:08.192070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"estimator=clf.best_estimator_\nestimator.score(X_test,y_test)","metadata":{"_uuid":"be99cf9bccf7f58d15af99449f03cfaec2913685","execution":{"iopub.status.busy":"2022-02-11T04:20:08.194321Z","iopub.execute_input":"2022-02-11T04:20:08.194665Z","iopub.status.idle":"2022-02-11T04:20:08.210542Z","shell.execute_reply.started":"2022-02-11T04:20:08.194609Z","shell.execute_reply":"2022-02-11T04:20:08.209826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Importance\n\nFeature importance is the most basic and widely used technique for model interpretation. We will also start with this.\n\nA  feature is “important” if shuffling its values increases the model error, because in this case the model relied on the feature for the prediction. A feature is “unimportant” if shuffling its values leaves the model error unchanged, because in this case the model ignored the feature for the prediction.","metadata":{"_uuid":"067a7580a2f954ba617a852fef1b55d58763756e"}},{"cell_type":"code","source":"import eli5\n# create our dataframe of feature importances\nfeat_imp_df = eli5.explain_weights_df(estimator, feature_names=features)\nfeat_imp_df","metadata":{"_uuid":"35e0fed670534661637517c584056cefc81577e3","execution":{"iopub.status.busy":"2022-02-11T04:20:08.211704Z","iopub.execute_input":"2022-02-11T04:20:08.212167Z","iopub.status.idle":"2022-02-11T04:20:08.646413Z","shell.execute_reply.started":"2022-02-11T04:20:08.212114Z","shell.execute_reply":"2022-02-11T04:20:08.645361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature=[\"Pregnancies\",\"Glucose\",\"BP\",\"SkinThickness\",\"Insulin\",\"BMI\",\"DPFunc\",\"Age\"]","metadata":{"_uuid":"61685871874873ffe13adefdac91080f8e9de412","execution":{"iopub.status.busy":"2022-02-11T04:20:08.647715Z","iopub.execute_input":"2022-02-11T04:20:08.648129Z","iopub.status.idle":"2022-02-11T04:20:08.652193Z","shell.execute_reply.started":"2022-02-11T04:20:08.647905Z","shell.execute_reply":"2022-02-11T04:20:08.651176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_feat_imp_df = pd.DataFrame(data=[tree.feature_importances_ for tree in \n                                     estimator],\n                               columns=feature)\n(sns.boxplot(data=all_feat_imp_df)\n        .set(title='Feature Importance Distributions',\n             ylabel='Importance'));","metadata":{"_uuid":"ee32ff5e8420829c944da6774eb1113688bc584b","execution":{"iopub.status.busy":"2022-02-11T04:20:08.653728Z","iopub.execute_input":"2022-02-11T04:20:08.654099Z","iopub.status.idle":"2022-02-11T04:20:09.205937Z","shell.execute_reply.started":"2022-02-11T04:20:08.654035Z","shell.execute_reply":"2022-02-11T04:20:09.205060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As per feature importance , the Glucose level in the blood along with BMI and age are the most important features in determining a diabetic patient. The result seem justified to me. High glucose level in blood is basically diabetes and obese people are more prone to it.\nOlder adults are at high risk for the development of type 2 diabetes due to the combined effects of increasing insulin resistance and impaired pancreatic islet function with aging.\n\nThese reults can also  be confirmed from the [Diabetes Care Website](http://care.diabetesjournals.org/content/35/12/2650)\n\nUptil now , we can say our model is doing a good job in classifying.It has learned the right weights and can be trusted.\n\nTakeaways:-\n* It  provides a highly compressed, global insight into the model’s behavior.\n* The importance measure automatically takes into account all interactions with other features. By permuting the feature you also destroy the interaction effects with other features.\n* Permutation feature importance is linked to the error of the model. In some cases, you might prefer to know how much the model’s output varies for a feature without considering what it means for performance\n*  Adding a correlated feature can decrease the importance of the associated feature by splitting the importance between both features.","metadata":{"_uuid":"4b7c9c2bb1be167419f9e1ec4a985a09da83c87d"}},{"cell_type":"code","source":"!pip install pydotplus","metadata":{"_kg_hide-output":true,"_uuid":"629269bbd7822b8d8d7a5c9e8448052788a05270","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-11T04:20:09.207122Z","iopub.execute_input":"2022-02-11T04:20:09.207380Z","iopub.status.idle":"2022-02-11T04:20:18.965709Z","shell.execute_reply.started":"2022-02-11T04:20:09.207329Z","shell.execute_reply":"2022-02-11T04:20:18.964777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's intuitive and fun to take a look at the tree.","metadata":{"_uuid":"1e95f787ef938028fcfba76ec656d5423d0cb847"}},{"cell_type":"code","source":"from IPython.display import Image  \nfrom sklearn.tree import export_graphviz\nimport graphviz\nimport pydotplus\nfrom io import StringIO  \n\n# Get all trees of depth 3 in the random forest\ndepths3 = [tree for tree in estimator.estimators_ if tree.tree_.max_depth==3]\n# grab the first one\ntree = depths3[0]\n# plot the tree\ndot_data = StringIO()\nexport_graphviz(tree, out_file=dot_data, feature_names=features, \n                filled=True, rounded=True, special_characters=True)\ngraph = pydotplus.graph_from_dot_data(dot_data.getvalue())  \nImage(graph.create_png())","metadata":{"_uuid":"4d66fac6a1f2a664be30fddcca8105c5c1b1102c","execution":{"iopub.status.busy":"2022-02-11T04:20:18.967139Z","iopub.execute_input":"2022-02-11T04:20:18.967449Z","iopub.status.idle":"2022-02-11T04:20:19.290672Z","shell.execute_reply.started":"2022-02-11T04:20:18.967394Z","shell.execute_reply":"2022-02-11T04:20:19.289818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Every non-leaf node is split based on the feature that is written at the top of the node. Towards the left part of the tree we classify samples as non-diabetic and towards the right part as diabetic. The entropy function at the leftmost leaf node becomes 0 because the data becomes homogenous(all the samples are either diabetic or non-diabetic). The first value in the value array tells how many samples are classified as non-diabetic and second value tells how many samples are diabetic.\n\nFor the leftmost leaf node , entropy is 0 all the 40 samples are non-diabetic.","metadata":{"_uuid":"e87c30b72a4d1cf363f5dc09d18bcef8d9bec6fa"}},{"cell_type":"markdown","source":"# Feature Interactions","metadata":{"_uuid":"f78e93eb49139856fd1fe98d4dec564c0370279d"}},{"cell_type":"markdown","source":"When features interact with each other in a prediction model, the prediction cannot be expressed as the sum of the feature effects, because the effect of one feature depends on the value of the other feature. The interaction between two features is the change in the prediction that occurs by varying the features after considering the individual feature effects.\n\nThe **H-statistic** proposed by Friedman and Popescu is used to calculate the interaction between features. There are a lot of packages in R for implementing this. Unfortunately for python users there is only one sklearn-gbmi package(to the best of my knowledge) to calculate H-statistic for Gradient-boosting models. ","metadata":{"_uuid":"62943ad6c982a34f7ade217f9e4bbd7f32fb045a"}},{"cell_type":"code","source":"!pip install sklearn-gbmi","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"_uuid":"8252b82d739d2e8dcbcd332dc6f5a4283fee3f38","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, let's implement  a Gradient-Boosting Classifier on our data first.","metadata":{"_uuid":"ce67af6ad3d217cbffca6520388c914479d123c7"}},{"cell_type":"code","source":"from sklearn.ensemble import GradientBoostingClassifier\ngbr_1 = GradientBoostingClassifier(random_state = 2589)\ngbr_1.fit(X_train,y_train)","metadata":{"_uuid":"809d2567a43a0c20c696cfdc8aafbaf652fcc6d5","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn_gbmi import *\nd=h_all_pairs(gbr_1,X_train)# d is a dictionary of feature pairs and their respective interaction strength\nl=sorted(d.items(), key=lambda x: x[1])#converted to a list sorted by interaction values","metadata":{"_uuid":"f759abc9d81bb5a3e11f97002a5b17b39f8be610","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"l=l[-10:] # let's just take the top 10\ndata=pd.DataFrame(l)\ndata.columns=['Feature',\"Interaction\"]\ndata.index=data['Feature']\ndata=data.drop(labels=['Feature'],axis=1)","metadata":{"_uuid":"a412250c3a9f69a7fd5663ac3b06d4cbd7c1300c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.plot(kind='barh', color='teal', title=\"Feature Interaction Strength\")","metadata":{"_uuid":"87837f1af89274dea95e18c678bef63b48206f0a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a strong interaction between number of pregnancies and age and slso between blood pressure and Insulin. All of these interactions are 2-way.\n\nTakeaways:-\n* The statistic detects all kinds of interactions, regardless of their particular form.\n* Since the statistic is dimensionless and always between 0 and 1, it is comparable across features and even across models.(not for python users)\n* The H-statistic tells us the strength of interactions, but it does not tell us how the interactions look like. That is what ***partial dependence plots*** are for.\n","metadata":{"_uuid":"561394620c6af1e775afbfe04ed41f86d4c59b3d"}},{"cell_type":"markdown","source":"# Partial Dependence Plots (PDP)\n","metadata":{"_uuid":"67222bc4ed4fa8a0ce67a5b023c838976de6ffb3"}},{"cell_type":"markdown","source":"The partial dependence plot (short PDP or PD plot) shows the marginal effect one or two features have on the predicted outcome of a machine learning model.It can show whether the relationship between the target and a feature is linear, monotonous or more complex.\n\nThe partial dependence plot is a both a global and local method: The method considers all instances and gives a statement about the global relationship of a feature with the predicted outcome (through the yellow line) and the relationship of all the unique instances with the outcome with the blue lines.","metadata":{"_uuid":"188ecd8a4ce10378785d96955e76a40221501211"}},{"cell_type":"code","source":"from pdpbox import pdp, info_plots\npdp_ = pdp.pdp_isolate(\n    model=estimator, dataset=X_train, model_features=features, feature='Glucose'\n)\nfig, axes = pdp.pdp_plot(\n    pdp_isolate_out=pdp_, feature_name='Glucose', center=True, \n     plot_lines=True, frac_to_plot=100\n)","metadata":{"_uuid":"268a1907a7e9d0a20e6c78a267ed017a2e92e0e1","execution":{"iopub.status.busy":"2022-02-11T04:20:19.291774Z","iopub.execute_input":"2022-02-11T04:20:19.291989Z","iopub.status.idle":"2022-02-11T04:20:20.586947Z","shell.execute_reply.started":"2022-02-11T04:20:19.291954Z","shell.execute_reply":"2022-02-11T04:20:20.585943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The y axis is interpreted as change in the prediction from what it would be predicted at the baseline or leftmost value.The blue lines are all the instances and the yellow line provides average marginal effect over them.The heterogenous effects can be seen by the blue lines.\n\nHigher blood sugar increases the chances of having diabetes.100 mg/dL is the mean  glucose level for non-diabetic people, which is also justified by the graph.","metadata":{"_uuid":"a163461da75870f8770cc55263524efb2649fbb1"}},{"cell_type":"code","source":"pdp_ = pdp.pdp_isolate(\n    model=estimator, dataset=X_train, model_features=features, feature='BMI'\n)\nfig, axes = pdp.pdp_plot(\n    pdp_isolate_out=pdp_, feature_name='BMI', center=True, x_quantile=True, \n     plot_lines=True, frac_to_plot=100\n)","metadata":{"_uuid":"b7be0b81a40e1727a3f085b21d81138effe41b97","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Obesity is related to diabetes, for BMI greater than 26 the chance of having diabetes goes on increasing with a greater effect.","metadata":{"_uuid":"875fa2a9f4975e58292e32f3b97da330f7741bd4"}},{"cell_type":"code","source":"pdp_ = pdp.pdp_isolate(\n    model=estimator, dataset=X_train, model_features=features, feature='Age'\n)\nfig, axes = pdp.pdp_plot(\n    pdp_isolate_out=pdp_, feature_name='Age', center=True, x_quantile=True, \n     plot_lines=True, frac_to_plot=100\n)","metadata":{"_uuid":"659e27e5755c121dc09837fcf610af2327a1de44","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After 23 years of age people are more suspectible to diabetes.","metadata":{"_uuid":"ca837d9d48c848d17126f40fe1765fae2f25fda1"}},{"cell_type":"markdown","source":"## 2D PDP plot\n\nThrough the Feature interaction plot , we found that age and pregnancies have strong interaction , let's see how they interact.","metadata":{"_uuid":"502c9278e8be75c9cc1478b8ab27e13d87468550"}},{"cell_type":"code","source":"!pip install pdpbox==0.1.0","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"from pdpbox import pdp, info_plots\nfeatures_to_plot = ['Age', 'Pregnancies']\ninter1  =  pdp.pdp_interact(model=estimator, dataset=X_train, model_features=features)\n\npdp.pdp_interact_plot(pdp_interact_out=inter1, feature_names=features_to_plot, plot_type='contour')\nplt.show()\"\"\"\n#the pdp_interact function seems to have some probelm with versioning, I'll uncomment this when it starts working","metadata":{"_uuid":"9e4968537db14ac8be3984bcefbb7ee335d550eb","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://i.imgur.com/d4b9U7g.png)\n\nWomen above 40 years of age with more than 7 pregnancies are most suspectible to the disease.","metadata":{"_uuid":"b67d666e916bd8da14f4b3348cab4f125197c5be"}},{"cell_type":"markdown","source":"Takeaways:-\n* Easy to implement and intuitive.\n* The assumption of independence is the biggest issue with PD plots. It is assumed that the feature(s) for which the partial dependence is computed are not correlated with other features. Which is never the case for real data.","metadata":{"_uuid":"29d65e015a556dea3467e196a315c9dd9783f3b1"}},{"cell_type":"markdown","source":"# Local Interpretable Model-agnostic Explaination (LIME)","metadata":{"_uuid":"39579934c490f13a76c332514120295906dad849"}},{"cell_type":"markdown","source":"LIME is a concrete implementation of local surrogate models.Surrogate models are trained to approximate the predictions of the underlying black box model. Local surrogate models are interpretable models that are used to explain individual predictions of black box machine learning models.\n\nLet's take a look at the steps:-\n1. Permute data.\n2. Calculate distance between permutations and original observation.\n3. Make predictions on the new data using the black-box model.\n4. Pick m features best describing the complex model outcome from the permuted data.\n5. Fit a simple(surrogate) model to the permuted data with m features and similarity score as weights.\n6. Feature weight from the surroagate model make explainations for the black box model's local behaviour.\n\n![](https://cdn-images-1.medium.com/max/800/1*vE3PUuhG6RRgK1J9oxg0nA.png)\n\nThe black-box model’s complex decision function f (unknown to LIME) is represented by the blue/pink background, which cannot be approximated well by a linear model. The bold red cross is the instance being explained. LIME samples instances, gets predictions using f, and weighs them by the proximity to the instance being explained (represented here by size). The dashed line is the learned explanation that is locally (but not globally) faithful.","metadata":{"_uuid":"7d74e38b0e60a6d8b8bb11104d62e4b852b7a436"}},{"cell_type":"code","source":"import lime\nimport lime.lime_tabular\nexplainer = lime.lime_tabular.LimeTabularExplainer(X_train.astype(int).values,  \nmode='classification',training_labels=y_train,feature_names=features,class_names=classes)\n#Let's take a look for the patient in 100th row\ni = 100\nexp = explainer.explain_instance(X_train.loc[i,features].astype(int).values, estimator.predict_proba, num_features=5)","metadata":{"_uuid":"7a67ea0ae90c242942adc83ac800be2e8a7420c3","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp.show_in_notebook(show_table=True)","metadata":{"_uuid":"c3cd9d581fe4c6b56673ba7872e864310c790c9c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Orange colored features supports diabetic class, and blue supports non-diabetic class.\n\nThere are three parts to the explanation :-\n1. The top Left part gives the prediction probabilities for class 0 and class 1.\n2. The right part gives the 5 most important features. Orange features support the diabetic class and blue support the non-diabetic class.\n3. The bottom part follows the same colour coding as 1 and 2. It contains the actual values  for the top 5 variables.\n\nThis can be read as *the woman is diabetic with a probability of 0.67. Her Glucose level, BMI , Age and DiabetesPedigreeFunction all add up to the prediction of diabetic and we have seen in the pdp plot how it does so. However , she has only one pregnancy which not at all contributes to diabetes but this has a lesser weight as compared to other more crucial features in determining diabetes*\n\nTo dig  a little deeper into  the implementation code of LIME , you can read the  [lime documentation](https://lime-ml.readthedocs.io/en/latest/lime.html#module-lime.lime_image)","metadata":{"_uuid":"8134fd867d8a1d142c778d5fb4938a91f476f82f"}},{"cell_type":"markdown","source":"## Submodular Pick (SP-LIME) for explaining models\n\nLIME aims to attribute a model’s prediction to human understandable features. In order to do this we need to run the explanation model on a diverse but representative set of instances to return a non redundant explanation set that is a global representation of the model. ","metadata":{"_uuid":"cf817ebeff386cfbbab4612c51eaee43b6a727a9"}},{"cell_type":"code","source":"from lime import submodular_pick\n# SP-LIME returns exaplanations on a sample set to provide a non redundant global decision boundary of original model\nsp_obj = submodular_pick.SubmodularPick(explainer, X_train.values, estimator.predict_proba, num_features=5,num_exps_desired=5)\n","metadata":{"_kg_hide-output":true,"_uuid":"cfce7e8c08ff73adace6c8e077256376bbe703f3","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"[exp.show_in_notebook() for exp in sp_obj.sp_explanations]","metadata":{"_uuid":"ff061f44d25c6ce30fd2d3f56cf7fbb7691e3bf5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Takeaways:-\n* Human-friendly explainations that are very useful when explaining to a lay person.\n* The explanations created with local surrogate models can use other features than the original model. This can be a big advantage over other methods, especially if the original features cannot bet interpreted.\n* A really big problem is the instability of the explanations. If you repeat the sampling process, then the explantions that come out can be different.\n","metadata":{"_uuid":"d9f49a6e6aa77f2cfd248593a1ad0e5ff6d20e23","trusted":true}},{"cell_type":"markdown","source":"# SHAP\n\nSHAP (SHapley Additive exPlanations)  can be used for both global and local explainations. It leverages game theory to help measure the impact of the features on the predictions .A prediction can be explained by assuming that each feature value of the instance is a *“player”* in a game where the prediction is the *payout*. The Shapley value  tells us how to fairly distribute the “payout” among the features.\n\nPlayers? Game? Payout? What is the connection to machine learning predictions and interpretability? The “game” is the prediction task for a single instance of the dataset. The “gain” is the actual prediction for this instance minus the average prediction for all instances. The “players” are the feature values of the instance that collaborate to receive the gain (= predict a certain value).\n\nLet's understand this with an example:\n\n![](https://christophm.github.io/interpretable-ml-book/images/shapley-instance.png)\n\nYou have trained a machine learning model to predict apartment prices. For a certain apartment it predicts €300,000 and you need to explain this prediction. The apartment has a size of 50 m2, is located on the 2nd floor, has a park nearby and cats are banned.\n\nThe average prediction for all apartments is €310,000. How much has each feature value contributed to the prediction compared to the average prediction?\n\nThe answer could be: The park-nearby contributed €30,000; size-50 contributed €10,000; floor-2nd contributed €0; cat-banned contributed -€50,000. The contributions add up to -€10,000, the final prediction minus the average predicted apartment price.\n\nNow you must be thinking ,how do we calculate the Shapley value for one feature?\n\nThe Shapley value is the average marginal contribution of a feature value across all possible coalitions. Coalitions are nothing but different simulated environments created by varying the feature and noticing the effect. For example if everything is kept the same and \" cat-banned\" is changed to \"cat-allowed\" , we check how the prediction changed. For more info , you can read [this](https://christophm.github.io/interpretable-ml-book/shapley.html).\n\nThis is taken from DanB's tutorial kernel.Let's do this for the 7th instance in our dataset.","metadata":{"_uuid":"93e8bb32c4c8440ce28293154e4d7414997f4b32"}},{"cell_type":"code","source":"import shap\n\n# create our SHAP explainer\nshap_explainer = shap.TreeExplainer(estimator)\n# calculate the shapley values for our data\nshap_values = shap_explainer.shap_values(X_train.iloc[7])","metadata":{"_uuid":"3f9f059585eea81e77b2f0fa200caaa44cff5d21","execution":{"iopub.status.busy":"2021-11-10T14:02:21.574188Z","iopub.execute_input":"2021-11-10T14:02:21.574779Z","iopub.status.idle":"2021-11-10T14:02:22.836007Z","shell.execute_reply.started":"2021-11-10T14:02:21.574714Z","shell.execute_reply":"2021-11-10T14:02:22.83517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load JS in order to use some of the plotting functions from the shap\n# package in the notebook\nshap.initjs()\nshap.force_plot(shap_explainer.expected_value[1], shap_values[1], X_train.iloc[7])","metadata":{"_uuid":"d9f2e624d2adf5e47e1b701203fb423a5cc3feea","execution":{"iopub.status.busy":"2021-11-10T14:02:34.441426Z","iopub.execute_input":"2021-11-10T14:02:34.441753Z","iopub.status.idle":"2021-11-10T14:02:34.469467Z","shell.execute_reply.started":"2021-11-10T14:02:34.4417Z","shell.execute_reply":"2021-11-10T14:02:34.468569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train.iloc[7]","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:03:06.969599Z","iopub.execute_input":"2021-11-10T14:03:06.969944Z","iopub.status.idle":"2021-11-10T14:03:06.978551Z","shell.execute_reply.started":"2021-11-10T14:03:06.969892Z","shell.execute_reply":"2021-11-10T14:03:06.977875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Features causing increase in prediction are in pink and features causing a decrease in prediction is in blue, along with their value showing the magnitude of effect. The base value is 0.3498 and we predict 0.7. This person is classified as diabetic , the features that pushed the result towards diabetic were Glucose level=161, Age=47, Insulin=132 and 10 pregnancies. The BMI feature which is low tries to negate the effect but couldn't because the combined effect of the pink features far outweighs it.\n\nIf you subtract the length of the blue bars from the length of the pink bars, it equals the distance from the base value to the output.","metadata":{"_uuid":"eb08255eae89b90a5b3139325373fe4f92d1c66b"}},{"cell_type":"markdown","source":"Let's also plot the **Summary plot** to get a global overview.","metadata":{"_uuid":"4258237428aa507a5bf9683479df099e3cd45124"}},{"cell_type":"code","source":"shap_values = shap_explainer.shap_values(X_train)\nshap.summary_plot(shap_values[1], X_train,auto_size_plot=False)","metadata":{"_uuid":"78437c16613b0e24ee94bb97f0281d2bc47eec90","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Okay, How to interpret this?\nThis plot is made of many dots. Each dot has three characteristics:\n\n* Vertical location shows what feature it is depicting\n* Color shows whether that feature was high or low for that row of the dataset\n* Horizontal location shows whether the effect of that value caused a higher or lower prediction.\n\nThe dots on the rightmost in the glucose row are pink which means Glucose level is high, which increases the chance of diabetes, we have already seen this before. \n\n***The feature insulin is not very clear to me , whether it's the natural insulin level in the patient or the amount of insulin artificially given to the patient.*** It shows a very unexpected behavior, it's high value can both increase, decrease the chance of having diabetes, which feels like a pretty useless insight to me.  Any input from your side regarding this will be  helpful !","metadata":{"_uuid":"e89090454246378b8c2914829f3b4fb78bdf4cfd"}},{"cell_type":"markdown","source":"# LIME for Text\n\nHere I'll be playing with LIME as discussed above but for text data.LIME for text differs from LIME for tabular data. Variations of the data are generated differently: Starting from the original text, new texts are created by randomly removing words from the original text. The dataset is represented with binary features for each word. A feature is 1 if the corresponding word is included and 0 if it has been removed.\n\nI'm using the comment classification dataset from the jigsaw toxic comment classification challenge. The model I train is very basic and the interpretations you see  maybe erroneous, but it will be a fault of our naive-model.\n","metadata":{"_uuid":"cd2b0fcf41539b51dea6f062ea14d68788905648"}},{"cell_type":"code","source":"train_zip=zipfile.ZipFile(\"../input/jigsaw-toxic-comment-classification-challenge/train.csv.zip\")\ntrain = pd.read_csv(train_zip.open('train.csv')).fillna(' ')\ntest_zip=zipfile.ZipFile(\"../input/jigsaw-toxic-comment-classification-challenge/test.csv.zip\")\ntest = pd.read_csv(test_zip.open('test.csv')).fillna(' ')\n\n\ntrain_text = train['comment_text']\ntest_text = test['comment_text']\nall_text = pd.concat([train_text, test_text])","metadata":{"_uuid":"ff921c4c13e2e81ff01d6d01aab1df8769de8205","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_vectorizer = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents='unicode',\n    analyzer='word',\n    token_pattern=r'\\w{1,}',\n    stop_words='english',\n    ngram_range=(1, 1),\n    max_features=10000)\nword_vectorizer.fit(all_text)\ntrain_word_features = word_vectorizer.transform(train_text)\ntest_word_features = word_vectorizer.transform(test_text)","metadata":{"_uuid":"63d89e369aa5b8d270b17240f33c1b76c6980c11","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First let's train our model for classifying toxic coments and we'll see with the help of LIME text explainer how it does so.","metadata":{"_uuid":"5ccd3dd33b582772e67728b0a1076be9aaf2d3ba"}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\ntrain_target_toxic = train['toxic']\nclassifier_toxic = LogisticRegression(C=0.1, solver='sag')\nclassifier_toxic.fit(train_word_features, train_target_toxic)","metadata":{"_uuid":"e21c5fb2ad695efbb0d509fbb0b0d6ca4468eb77","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"names=['non-toxic','toxic']","metadata":{"_uuid":"28b37f9cc6eb4d1fce9096ff006990fcacb0d69c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.pipeline import make_pipeline\nfrom lime.lime_text import LimeTextExplainer\nc_tf = make_pipeline( word_vectorizer,classifier_toxic)\nexplainer_tf = LimeTextExplainer(class_names=names)","metadata":{"_uuid":"c288952f5a77a9b77ed1888e8e23f2f04253520a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp = explainer_tf.explain_instance(train_text.iloc[802], c_tf.predict_proba, num_features=4, top_labels=1)\nexp.show_in_notebook(text=train_text.iloc[802])","metadata":{"_uuid":"71e48fd66dd5372c38aad50dec3f1301396c2ae7","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp = explainer_tf.explain_instance(train_text.iloc[55], c_tf.predict_proba, num_features=4, top_labels=1)\nexp.show_in_notebook(text=train_text.iloc[55])","metadata":{"_uuid":"6840383bbe4bb6ca24b50ec49172a7cecb06beea","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's see the same for threatening comments.","metadata":{"_uuid":"e7eeedcf53edf2825fbe81bd77f80f9adf16a43a"}},{"cell_type":"code","source":"train_target_threat = train['threat']\nclassifier_threat = LogisticRegression(C=0.1, solver='sag')\nclassifier_threat.fit(train_word_features, train_target_threat)","metadata":{"_uuid":"ec00b9effcb84d1c47f5e0abd898096c38f3bd01","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"names=['threatening','non-threatening']\nc_tf = make_pipeline( word_vectorizer,classifier_threat)\nexplainer_tf = LimeTextExplainer(class_names=names)\n","metadata":{"_uuid":"fb0ac51766b727b543ae4131f20516adb4713cb1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp = explainer_tf.explain_instance(train_text.iloc[79], c_tf.predict_proba, num_features=4, top_labels=1)\nexp.show_in_notebook(text=train_text.iloc[79])","metadata":{"_uuid":"aaf50b1aaf6d6a2af0b420d04ea7f06c24201745","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp = explainer_tf.explain_instance(train_text.iloc[1085], c_tf.predict_proba, num_features=4, top_labels=1)\nexp.show_in_notebook(text=train_text.iloc[1085])","metadata":{"_uuid":"44d5c59b8164380289e7627db41a6c9fb33b3104","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LIME for Images\n\nNow comes probably the most interesting part, **interpreting images classified by a neural network**. LIME for images works differently than LIME for tabular data and text. Intuitively, it would not make much sense to perturb individual pixels, since many more than one pixel contribute to one class. Randomly changing individual pixels would probably not change the predictions by much. \n\nTherefore, variations of the images are created by segmenting the image into “superpixels” and turning superpixels off or on. Superpixels are interconnected pixels with similar colors and can be turned off by replacing each pixel with a  color such as gray.\n\nI'm using a pretrained Inception V3 model.","metadata":{"_uuid":"d8c501af6e5e82ebbbc35c44848a97245043662b"}},{"cell_type":"code","source":"import keras\nfrom keras.applications import inception_v3 as inc_net\nfrom keras.preprocessing import image\nfrom keras.applications.imagenet_utils import decode_predictions\nfrom skimage.io import imread","metadata":{"_uuid":"429a2d952e49fc5a3b6cd472fb5c7af6fb111c10","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inet_model = inc_net.InceptionV3()#load the model for prediction","metadata":{"_kg_hide-output":true,"_uuid":"22ed00f91a5687bbe2831094ffbaf8413e8c9c84","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def transform_img_fn(path_list):\n    out = []\n    for img_path in path_list:\n        img = image.load_img(img_path, target_size=(299, 299))\n        x = image.img_to_array(img)\n        x = np.expand_dims(x, axis=0)\n        x = inc_net.preprocess_input(x)\n        out.append(x)\n    return np.vstack(out)","metadata":{"_uuid":"d66f526b9f3470b8227b3251aa5b9f265154822e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"main_dir=\"../input/inception-classification-sample-images/inception classification samples/inception classification samples\"","metadata":{"_uuid":"9158cd08ce002f09797e0a590369e47cad9478c5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images = transform_img_fn([os.path.join(main_dir,'cat-and-mouse.jpg')])\n# I'm dividing by 2 and adding 0.5 because of how this Inception represents images\nplt.figure(figsize=(3,3))\nplt.imshow(images[0] / 2 + 0.5)\npreds = inet_model.predict(images)\nfor x in decode_predictions(preds)[0]:\n    print(x)","metadata":{"_uuid":"a92628fa1c085ff061981aaa940359519c531733","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model predicts 5 classes with different probabilities. Time to get an explaination of the model. ","metadata":{"_uuid":"7724da33736583dd8529928234492cda8e463e2f"}},{"cell_type":"code","source":"import lime\nfrom lime import lime_image\nexplainer = lime_image.LimeImageExplainer()\nexplanation = explainer.explain_instance(images[0], inet_model.predict, top_labels=5, hide_color=0, num_samples=1000)","metadata":{"_uuid":"8080d41fd3311bb19c4964204939bc5017bd057f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see an explaination for classifying as tiger cat. The first parameter in the get_image_and_mask function is the label for the class. The class for us is tiger cat and you can find the corresponding label here [https://savan77.github.io/blog/files/labels.json](https://savan77.github.io/blog/files/labels.json). It is 282.We can see the top 8 superpixels that are most positive towards the class with the rest of the image hidden.","metadata":{"_uuid":"4006ecef0e4f3afe6e25d68ca5af258aa4076a10"}},{"cell_type":"code","source":"from skimage.segmentation import mark_boundaries\ntemp, mask = explanation.get_image_and_mask(282, positive_only=True, num_features=8, hide_rest=True)\nplt.figure(figsize=(3,3))\nplt.imshow(mark_boundaries(temp / 2 + 0.5, mask))","metadata":{"_uuid":"2732c38ee4a4f704b630597127fbf6f6610ea467","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's view it with the full image now.","metadata":{"_uuid":"8063b80b49b928e172870a8dfc6e393dd0262544"}},{"cell_type":"code","source":"temp, mask = explanation.get_image_and_mask(282, positive_only=True, num_features=8, hide_rest=False)\nplt.figure(figsize=(3,3))\nplt.imshow(mark_boundaries(temp / 2 + 0.5, mask))","metadata":{"_uuid":"09be10932482c1e3cfbd92aa7bf8c0218560a249","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The parameter **positive_only** , when True, only take superpixels that contribute to the prediction of the label. Otherwise, uses the top num_features superpixels, which can be positive or negative towards the label.","metadata":{"_uuid":"50e27015663f6b1ff0475507de1cacd4833b1b00"}},{"cell_type":"code","source":"import lime\nfrom lime import lime_image\nexplainer = lime_image.LimeImageExplainer()\nexplanation = explainer.explain_instance(images[0], inet_model.predict, top_labels=5)\ntemp, mask = explanation.get_image_and_mask(282, positive_only=False, num_features=100)\nplt.figure(figsize=(3,3))\nplt.imshow(mark_boundaries(temp / 2 + 0.5, mask))","metadata":{"_uuid":"8e47659731ab7a5e2b283cf83287f72545145b05","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The superpixels in green are positive towards the label i.e. tiger cat and superpixels in red are negatve.\n\nLet's now see the interpretations for class \"mouse\".The label for mouse is 673.","metadata":{"_uuid":"b36eff1cd9c20cde350e9d1ae849aeb33f4fb6da"}},{"cell_type":"code","source":"from skimage.segmentation import mark_boundaries\ntemp, mask = explanation.get_image_and_mask(673, positive_only=True, num_features=5, hide_rest=False)\nplt.figure(figsize=(3,3))\nplt.imshow(mark_boundaries(temp / 2 + 0.5, mask))","metadata":{"_uuid":"76c1b956e6ace32bf35ff308ccbc714b73108c85","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's very clear from this image that the model is not doing a good job in predicting mouse. Since the superpixels used are of the cat mostly.","metadata":{"_uuid":"5bfba4b658b53fe007115e415bc8aecd741aabcb"}},{"cell_type":"markdown","source":"Thanks for reading guys, I'll be happy if you can upvote this kernel. I'm eager to hear any suggestions and feedback from your side.\n\nLoads of thanks to this amazing book [Interpretable Machine Learning](https://christophm.github.io/interpretable-ml-book/) by Christoph Molnar. Some other references used by me are:-\n\n* [https://www.kdnuggets.com/2018/06/human-interpretable-machine-learning-need-importance-model-interpretation.html](https://www.kdnuggets.com/2018/06/human-interpretable-machine-learning-need-importance-model-interpretation.html)\n* [http://savvastjortjoglou.com/intrepretable-machine-learning-nfl-combine.html](http://savvastjortjoglou.com/intrepretable-machine-learning-nfl-combine.html)\n* [https://lime-ml.readthedocs.io/](https://lime-ml.readthedocs.io/)\n* [https://github.com/marcotcr/lime](https://github.com/marcotcr/lime)","metadata":{"_uuid":"28a4c5b81d790a76c3fd15e0041800a417f9f823"}}]}