{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Evaluation of Machine Learning Models applied to Text Sentiment Analysis\n\nThe problem of making Machine Learning (ML) models as transparent as possible is receiving significant attention in the Data Sciece world. \nModel transparency is an important objective and attempts to achieve one or more of the following specific goals:\n\n### 1. Understand model behavior and drivers enough in order to debug or, even better, improve an existing model along dimensions of interest\n\nQuestions we usually ask in order to get to this goal are:\n\n- What are the key model drivers? How do they affect the model?\n\n- What are key interactions between individual model drivers?\n\n### 2. Figure out what is the range of application of an existing model (focus on implementation / performance monitoring)\n\n- What data does the model work well on and what data receives inconclusive predictions/results?\n\n- Often models lose their predictive power as time passes and the nature of the data changes. When the data mix collected for scoring is very different from the one at model development time predictions deteriorate. Can we predict how long a model can remain valid for?\n\n- What cases become outliers depending on modeling strategy?\n\n### 3. Understand why some cases fit well in the model while others are not (related to goal 2 but with focus on equity / impact disparity)\n\nIn a classification or regression setting some observations receive extreme estimated values and other average. In a clustering exercise, some observations belong clearly to clusters and others do not. Is it because this is the behavior of the underlying data? Or, is it that modeling choices favor/disfavor some classes of data?\n\n### 4. Explore areas for model improvement.\n\nWithin a given methodology, how can we pick model parameters that a) optimize model performance (based on a preselected performance metric), and/or b) improve convergence speed.","metadata":{}},{"cell_type":"markdown","source":"We base our analysis on a model that is relatively easy to evaluate addressing a Natural Language Processing (NLP) problem. The problem is described in detail in the Quora Insincere Questions Competition and relevant data are easy to obain from Kaggle. The ML method we use here is a tree ensemble method, Gradient Boosting. We use open-source software for this exercise, Python SciKit-Learn. This well-maintained computing environment permits us to combine packages smoothly. The model fit we obtain is not as good as what we get from strategies based on Deep Learning (DL), in particular multi-layer neural networks for sequence processing with embeddings, bidirectional architecture, and attention layers.\n\nWe apply a number of common tools for model evalution:\n\n- Feature Importance Analysis\n- Analysis of the first (most important) tree in the ensemble\n- In-depth analysis of selected documents, with emphasis on what our model misclassifies\n\nIn addition, we leverage 2 recently developed Python packages developed precisely to assist modelers in ML model interpretation. Both, packages provide tools specific to NLP models, which come in very handy for our task. They are:\n\n- Eli5, and\n- Shap","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom sklearn import tree\nfrom sklearn.pipeline import Pipeline, make_pipeline\n\n\n\nimport eli5\nfrom eli5.lime import TextExplainer\n\nimport shap","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are the data from Quora Insincere Questions Competition\n\nFor more details follow the link: https://www.kaggle.com/c/quora-insincere-questions-classification/overview","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/quora-insincere-questions-classification/train.csv\",index_col=0)\nprint( df.info() )\nprint('\\nTarget Level Counts:')\nprint(df.target.value_counts())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To get a feel of the questions at hand and of the target variable (insincere question), here are some question examples by label.","metadata":{}},{"cell_type":"code","source":"n_examples_by_target = 100\ntest_examples = df.groupby(\"target\").head(n_examples_by_target).sort_values(by='target')\n\nwith pd.option_context('display.max_colwidth', 400):\n    for t in [0,1]:\n        print(\"\\n{} Question Examples\\n\".format(['Sincere','Insincere'][t]))\n        for q in test_examples.loc[test_examples.target==t,'question_text'].iloc[:10]:\n            print(q)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Develop Text Processing Pipeline","metadata":{}},{"cell_type":"code","source":"max_features = 5000 # Max Number of Words / NGrams to Consider\nn_docs = 300000 # Number of Documents for Development and Validation\nclass_names = ['sincere','insincere']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Vectorizer Definition and Parameters\n# Parameter choices have significant impact and need to be optimized\n\nvectorizer = TfidfVectorizer(min_df=10, max_df=0.1, max_features=max_features)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# GB Classifier Definition and Parameters\n# Again parameter choices are very important. They impact model performance and computational cost/time.\n\nclf = GradientBoostingClassifier(n_estimators=100, max_depth=6, subsample=0.7, random_state=21)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The data is too big for the available computational resources, so we select a subset for development and validation.","metadata":{}},{"cell_type":"code","source":"np.random.seed(2021)\nidx = np.random.choice(len(df),n_docs)\nn_cut = int(n_docs*0.66)\ntrain_idx = idx[:n_cut]\ntest_idx = idx[n_cut:]\nprint(len(train_idx),len(test_idx))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_set = (df.iloc[train_idx,0].values, df.iloc[train_idx,1].values)\ntest_set  = (df.iloc[test_idx,0].values,  df.iloc[test_idx,1].values)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We need to be able to score documents in raw form. To make this easy we fit a pipeline that vectorizes and classifies.","metadata":{}},{"cell_type":"code","source":"pipe = make_pipeline(vectorizer, clf)\n\npipe.fit(*train_set)\npipe.score(*test_set)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fit is good, but not as good as what we get with DL models. See also F1_Score below.","metadata":{}},{"cell_type":"markdown","source":"#### Vectorizer Parameters\n\nSome were selected before, others get default values.","metadata":{}},{"cell_type":"code","source":"pipe[0].get_params()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = vectorizer.get_feature_names()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Words With High TFIDF Sum**","metadata":{}},{"cell_type":"code","source":"X = vectorizer.transform(test_set[0])\nprint(*sorted([w for w,x in zip(features,np.ravel(X.sum(0))) if x > 200]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note prevalence of India, Indians, 2017, Trump, Quora. These words capture the period (2017, Trump), an overrepresented demographic group (Indians), and words specific to this application (Quora). Naturally, we expect that as time goes by some of these words will lose their value as predictors and other might gain additional importance (e.g., Quora, Quorans).","metadata":{}},{"cell_type":"markdown","source":"#### Classifier Fit Evaluation","metadata":{}},{"cell_type":"markdown","source":"**OOB Improvement Analysis**","metadata":{}},{"cell_type":"code","source":"pd.Series(clf.oob_improvement_).plot(figsize=(12,7),drawstyle=\"steps\");","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model converged in 100 steps, reaching low error values after 30 rounds. Observe, the jump in the first step. The first tree in the ensemble captures about 40% of the explanatory power of the full ensemble.\n\nOur focus here is on ML model evaluation, so this model is not optimized. A different modeling strategy, restricting tree depth to a low value (i.e. permitting only shallow trees in the ensemble) might have resulted in a better fit, but the first tree would not capture as much from the essence of the ensemble.","metadata":{}},{"cell_type":"code","source":"y_pred = pipe.predict_proba(df.iloc[test_idx,0].values)[:,1] > 0.06\nprint(classification_report(df.iloc[test_idx,1].values, y_pred))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The performance is relatively good, but DL models, as mentioned earlier, have better performance and in particular better F1-score. The objective in this competition is to optimize the F1-score and the winners used DL models.","metadata":{}},{"cell_type":"markdown","source":"## Feature Importance","metadata":{}},{"cell_type":"code","source":"(\n    pd.DataFrame({\n        'feature': features,\n        'importance': clf.feature_importances_,\n    })\n    .sort_values('importance', ascending=False)\n).iloc[:30].set_index('feature')['importance'].plot.barh(figsize=(8,10));","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe that \"loaded\" words, e.g. \"Trump,\" \"muslims,\" \"gay,\" tend to top the list of most prevalent terms in the tree ensemble. We will remark in a following section below that other words, e.g. mathematical or scientific terminology, tend to be associated with sincere questions. The latter, though, are not at the top of the feature importance list.","metadata":{}},{"cell_type":"markdown","source":"## Overview of First Tree\n\nIt carries the most predictive power, close to half the one obtained by the full ensemble.\n\nSome prior exposure to Decision Trees permits us to understand how many terms combine to produce insincere questions. For instance, the combination of women-white-get leads to one of the highlighted in deep orange leafs below. A combination of muslims with south leads to a negative logit and therefore lower likelihood for a question with insincere label. This is the power of analyzing the first tree in an ensemble, if tree complexity is not too high (in this case max tree depth is 6). In contrast, all other methods below are not well suited to evaluate interactions of model drivers.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(55,15))\ntree.plot_tree(clf.estimators_[0][0],fontsize=12,filled=True, feature_names=features, label=None, proportion=True);","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(tree.export_text(clf.estimators_[0][0], feature_names=features))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Evaluation and Transparency\n\nWe focus on the examples we selected randomly earlier and pick cases that were misclassified by our model as well as some that were correctly classified. We try to develop insights about our model by examining these examples in detail. For consistentcy, the same examples will be analyzed below using the model transparency utilities Eli5 and Shap.","metadata":{}},{"cell_type":"code","source":"ex_score = pipe.predict_proba(test_examples.question_text.values)[:,1]\npd.Series(ex_score).describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_sincere   = sorted([\"{:.3f}|{:3d}\".format(s,i) for i, s in enumerate(ex_score[:n_examples_by_target])], reverse=True)\nex_insincere = sorted( \"{:.3f}|{:3d}\".format(s,i) for i, s in enumerate(ex_score[n_examples_by_target:],start=100))\nprint('Sincere Examples')\nprint(\" With relatively high probability of being insincere\")\nprint(*ex_sincere[:5],sep=\"\\n\")\nprint(\" Correctly estimated\")\nprint(*ex_sincere[-5:],sep=\"\\n\")\nprint('\\n\\nInsincere Examples')\nprint(\" Incorrectly estimated\")\nprint(*ex_insincere[:5],sep=\"\\n\")\nprint(\" Correctly estimated\")\nprint(*ex_insincere[-5:],sep=\"\\n\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exid = [65, 51, 41, 103, 104, 140]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Preprocess and Tokenize Just as in Development Time\nvpre = vectorizer.build_preprocessor()\nvtok = vectorizer.build_tokenizer()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**FI Color Scale**\n\nWords not in model are not highlighted","metadata":{}},{"cell_type":"code","source":"pd.Series(np.cumsum(sorted(clf.feature_importances_))).plot(figsize=(12,7),drawstyle=\"steps\",grid=True,title=\"Feature Importance Cumulative Sum\");","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"word_fi = clf.feature_importances_[clf.feature_importances_>1e-3]\nprint(\"Top {:3d} Words Capture {:5.1%} of Feature Importance\".format(len(word_fi), word_fi.sum()))\nword_fi2 = word_fi[word_fi>0.5e-2]\nprint(\"Top {:3d} Words Capture {:5.1%} of Feature Importance\".format(len(word_fi2), word_fi2.sum()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that only 140 out of 5,000 words capture 88.5% of total feature importance. 38 words capture 2/3 of feature importance.","metadata":{}},{"cell_type":"code","source":"fi_bins = [0,1e-3,5e-3,1e-2,1]\nfid = np.digitize(clf.feature_importances_, fi_bins) - 1\nfi_intensity = {w:k for k,w in zip(fid,features)}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fi_word_color(w,i):\n    # grey, yellow, magenta, red\n    cmap = [97, 93, 95, 91]\n    return f'\\033[{cmap[i]};4m' + w + '\\033[0m'\n\nfor i, x in enumerate(zip(fi_bins,fi_bins[1:])):\n    print( fi_word_color(\"FI from {:.4f} to {:.4f}\".format(*x),i) )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def doc_highlight(doc,fn,intensity):\n    tokens = vtok(vpre(doc))\n    out = []\n    for w in tokens:\n        out.append( fn(w, intensity[w]) if w in intensity else w)\n    return ' '.join(out)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Highlight Documents Using Feature Importance","metadata":{}},{"cell_type":"code","source":"for id in exid:\n    print(\"True Class:\",class_names[test_examples.target[id]],\"Score={:.4f}\".format(ex_score[id]))\n    print(doc_highlight(test_examples.question_text.values[id],fi_word_color,fi_intensity))\n    print()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Red words tend to be the biggest drivers as far as our fitted model is concerned. \n\n- In the second question, \"indians\" animates our model giving this question higher probability than it deserves.\n\n- In the fourth question, the word \"most\" is surprisingly the most meaningful by our model. The string of insults \"overweight misfit homeschooled bimbo\" is not even underlined, as these terms do not even belong to the model vocabulary of top 5,000 words. Looking at this, a modeling strategy that naturally suggests itself is to increase our model's vocabulary by enhancing the development sample and our investment in computational resources.","metadata":{}},{"cell_type":"markdown","source":"**Probability Insincere Color Scale**\n\nWords not in model are not highlighted","metadata":{}},{"cell_type":"code","source":"p_num = pipe.predict_proba([''])[0][1] # Use as numeraire the probability insincere associated with any UNKNOWN word if taken in isolation\npi_bins = [0,p_num,2*p_num,3*p_num,4*p_num,1]\npid = np.digitize(pipe.predict_proba(features)[:,1], pi_bins) - 1\npi_intensity = {w:k for k,w in zip(pid,features)}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pi_word_color(w,i):\n    # grey, yellow, magenta, red\n    cmap = [42,46,47,43,41]\n    return f'\\x1b[5;39;{cmap[i]}m' + w + '\\x1b[0m'\n\nfor i, x in enumerate(zip(pi_bins,pi_bins[1:])):\n    print( pi_word_color(\"FI from {:.4f} to {:.4f}\".format(*x),i) )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Highlight Documents Using Invidual Word Probability","metadata":{}},{"cell_type":"code","source":"for id in exid:\n    print(\"True Class:\",class_names[test_examples.target[id]],\"Score={:.4f}\".format(ex_score[id]))\n    print(doc_highlight(test_examples.question_text.values[id],pi_word_color,pi_intensity))\n    print()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In some questions we see emerging patterns using this analysis:\n\n- Question 2: Teal words combined with a red word (\"indians\") result in higher probability insincere for the question as a whole\n\n- Question 3: Teal words combined with a green word, carrying positive meaning by our model (\"best\"), result in low probability insincere for the question as a whole\n\n- Questions 4 & 5: many teal words without accompanying red/yellow words result in undeservedly low probability insincere for the question as a whole.","metadata":{}},{"cell_type":"markdown","source":"Both pictures (feature importance highlighting and probability highlighting) provide some helpful insights, but fail to provide insights about how drivers interact to increase or reduce the likelihood of a question being labeled insincere.","metadata":{}},{"cell_type":"markdown","source":"## 1. ELI5 \n\n### TextExplainer","metadata":{}},{"cell_type":"code","source":"for id in exid:\n    doc = test_examples.question_text.values[id]\n    print(\"True Class:\",class_names[test_examples.target[id]],\"Score={:.4f}\".format(ex_score[id]))\n    te = TextExplainer(random_state=42)\n    te.fit(doc, pipe.predict_proba)\n    display(te.show_prediction(target_names=class_names))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**These examples provide some interesting insights:**\n\n- First Question, \"Is it just me or have you ever been in this phase wherein you became ignorant to the people you once loved, completely disregarding their feelings/lives so you get to have something go your way and feel temporarily at ease. how did things change?\", is labeled \"sincere\" but prob insincere is 36%. It seems that the given label is arbitrary, so this analysis does help to debug the model and the data collection process.\n\n- Third Question, \"what are some best college for aircraft propulsion(m.s)?\", here the words \"college,\" \"aircraft,\" and \"propulsion\" appear to help the model classify the question correctly as sincere. Unfortunately, the word \"propulsion\" is not in the vocabulary of vectorized terms (it is coded as unknown word).\n\n- Fourth Question, \"what is the most degrading way to insult a grammar obsessed overweight misfit homeschooled bimbo?\". Looking at the text explainer one observes that the words 'way,\" \"grammar,\" and \"overweight,\" have a stronger effect than \"obsessed bimbo\" and result in the question being misclassified as \"sincere\". Many of these words (e.g., \"degrading,\" \"bimbo\"), though, have no relation to our model and they do not belong to the vocabulary of vectorized words.\n\n**Overall**\n\nThis tool creates its own default model and does not explain the model we developed.","metadata":{}},{"cell_type":"markdown","source":"#### Eli5 Feature Importance","metadata":{}},{"cell_type":"code","source":"x_res = vectorizer.transform(test_examples.question_text.values)\ny_res = test_examples.target.values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eli5.show_weights(clf, feature_names = features, top=50)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This tool is useful but seems to provide the same results with feature importance at a significantly higher computational cost. \n\nTo be fair Eli5 tries to provide insights even when model methodology is unknown.","metadata":{}},{"cell_type":"markdown","source":"An additional metric \"permuation importance\" has very high computational cost and we didn't evaluate it here.","metadata":{}},{"cell_type":"markdown","source":"# SHAP Analysis","metadata":{}},{"cell_type":"code","source":"x_res = vectorizer.transform(test_examples.question_text.values)\nx_res_df = pd.DataFrame(x_res.todense(), columns = features)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"explainer = shap.Explainer(clf)\n\nshap_values = explainer(x_res_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shap.initjs()\n\nfor id in exid:\n    print(\"True Class:\",class_names[test_examples.target[id]],\"Score={:.4f}\".format(ex_score[id]))\n    print(test_examples.iloc[id,0])\n    display(shap.plots.force(shap_values[id], feature_names=features))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Interesting graph, but the weights of each word in a sentence (Shapley values) appear arbitrary. Again, they seem to capture the sum of individual effects and not the effect of word interactions.","metadata":{}},{"cell_type":"markdown","source":"#### Shap's Alternatives of Feature Importance","metadata":{}},{"cell_type":"code","source":"shap.initjs()\nshap.plots.beeswarm(shap_values,20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This graph appears to capture the terms at the top of the feature importance list. Hence, it is useful. But, complex calculations need to take place to obtain this graph.","metadata":{}},{"cell_type":"code","source":"shap.plots.bar(shap_values, 20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This graph is an alternative view of the beeswarm plot.","metadata":{}},{"cell_type":"code","source":"shap.decision_plot(explainer.expected_value, shap_values.values, features, link='logit')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shap.plots.heatmap(shap_values, max_display = 20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again, the heatmap plot appears to show the words from the feature importance short-list, but assigns difficult to interpet weights to them. More important, we get no insights about how words interact to change a document's classification.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\n\nWe return to the model evaluation objectives in the beginning of this review and provide a summary of how well we did with our analysis. The results are not promissing either when one uses the standard methods (feature importance and top tree in ensemble) or when newer approaches, like Eli5 and Shap, are used.\n\n- Understand model behavior and drivers enough in order to debug or, even better, improve an existing model\n\nOur analysis does provide insights that help us understand the nature of the data at hand and the consistency of labels. Unfortunately, only top-tree analysis provides some insights about model driver interaction and this is only available and meaningful when using tree ensemble methods with a top tree that is interpretable and has high explanatory value. So, we need a top tree with not too many branches **and** explanatory value comparable to the one on the ensemble.\n\nAt the end of this exercise, our insights are little useful in improving the model (see model optimization below).\n\n- Figure out what is the range of application of an existing model\n\nSome insights are useful. Certain words reflect discussions in the period the data were collected (2017, Trump, Muslims). We expect that the model will lose its explanatory power now, in 2021. Some neutral words (e.g. \"southern\", \"college\") in contrast might retain their value. \n\nWe cannot see how one can provide answers about our dataset's range of application (a model expiration date if you will) that are significantly more detailed than what we reported in the previous sentence. Of course, when the model loses its predictive power, then we know its validity is in question, but we cannot predict in advance model deterioration, as we could have if we fully understood model structure. \n\n- Why do some cases achieve high importance (score) and others do not?\n\nML models are not transparent and we have encountered cases where positive and negative terms interact in ways that are difficult to understand. \n\nIn general, it is possible to establish with this analysis some types of systematic biases. For instance, this dataset shows that a lot of people from India like to participate in Quora and it seems that their perspectives are relatively more prevalent. It is difficult, however, to get an even deeper appreciation of which subgroups of the Indian population were represented in Quora circa 2017. \n\n- How can we optimize a machine learning model?\n\nThe typical way to optimize a ML model is to perform some sort of hyper-parameter optimization. This process is not much dependent on the results of the previous analysis. \n\nGiven insights from previous work on the same dataset, however, we have concluded that these Quora questions contain many spelling errors, have international characters, and may benefit from masking numbers (i.e., instead of using 134 and 182 as separate words combine them all as 3 digit numbers). All these insights might help model optimization, but require significant detective work and are not the outcome of a simple model evaluation process.\n\n*Overall assessment*: We considered a tree-ensemble model which does not perform as well as, say, DL models, but is relatively easier to interpret. After some reasonable steps at model evaluation we find that we know very little about how our model works or about the nature of our dataset. The Socratic saying \"the only true wisdom is in knowing you know nothing\" fits particularly well to machine learning modeling.","metadata":{}}]}