{"cells":[{"metadata":{"_uuid":"d088ad8cad761921f1b9040eb9322b62fd799e5a"},"cell_type":"markdown","source":"# NLP Capstone Project - Movie Review - Sentiment Analysis -  Classify the sentiment of sentences from the Rotten Tomatoes dataset\n## Developing Machine Learning Models using TF - IDF as feature extraction / representation\n\n![](https://cdn.steemitimages.com/DQmQZCf7ME7Haj3X3MzXtG8R8JtGmTpuh5NXDSd3wKueva7/rottentomatoes.png)\n\n![Sentiment](https://www.kdnuggets.com/images/sentiment-fig-1-689.jpg)\n\n![ML](https://juststickers.in/wp-content/uploads/2017/04/machine-learning.png)\n\n\nMoving one from EDA to applying Machine Leaning models with the movie review sentiment analysis dataset. Inparticular the ML models will be developed in conjunction with TF - IDF as feature representation. \n\nAt first, There must be mention that after EDA an odd conclusion was made. The dataset of this competition turned to have some unique features. we have only phrases as data. And a phrase can contain a single word. And one punctuation mark can cause phrase to receive a different sentiment. Also assigned sentiments can be strange. This means several things:\n\n- using stopwords can be a bad idea, especially when phrases contain one single stopword;\n- puntuation could be important, so it should be used;\n\n** This thought will be enhanced later with my anomaly detection insights **\n\n"},{"metadata":{"_uuid":"f5c933949f6eef7669ee8f554cab65bbcf456c15"},"cell_type":"markdown","source":"## Libraries\nLoading most important libraries for the rest of the Machine Learning Analysis."},{"metadata":{"trusted":true,"_uuid":"279572634fb1985999424d259fc038eb12732c9a"},"cell_type":"code","source":"import pandas as pd\n\nimport matplotlib\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\n\nimport nltk","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1079c746f344c1d127d797215681eee1c3727739"},"cell_type":"markdown","source":"### load the dataset"},{"metadata":{"trusted":true,"_uuid":"8ed18e14e5d260be8729fb4a815f570b66ed1c72"},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.tsv\", sep=\"\\t\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b59b9be7208c0aa71889397f8a3a7a4695cdd6a5"},"cell_type":"code","source":"df_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"509ecba55df17d0dc7f797aa2df25fb357f81bed"},"cell_type":"markdown","source":"### Preview"},{"metadata":{"trusted":true,"_uuid":"f1bc83fe85a53c3aab52fa91964cc9d2a7dcd5c3"},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5140913df00248ffa0926479de4b0217e562d9ae"},"cell_type":"code","source":"df_test.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a1a5c21f398844b84487e668755e452bf389b431"},"cell_type":"markdown","source":"### Thoughts before training ML models\nHere are couple of instances where punctuations appeared to be predictive. So if we \"cleanedup\" the data in the name of data preparation some predictiveness will be lost."},{"metadata":{"trusted":true,"_uuid":"67a87798e5a3e109df45bce313628487b0f6dc2f"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 0) & (df['PhraseId'] <= 2)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48d6686f0716d9729cc6fa056465366ee9964091"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 517) & (df['PhraseId'] <= 518)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8bd6b38053c393ea2984e5d1ee5ff1d283398b75"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \",\" is important"},{"metadata":{"trusted":true,"_uuid":"4ed70044592e5417db36d3842a48726b04640452"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 68) & (df['PhraseId'] <= 69)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3db985d9cc3ae2a00a3f5f0a5efcc801ed2fa759"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \"!\" is important"},{"metadata":{"trusted":true,"_uuid":"b574cd173e3d6d9e73877fe200dce6ed22c29d25"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 10737) & (df['PhraseId'] <= 10738)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f40ff093f9e04724f79abc722b04ceebd0f49ce1"},"cell_type":"markdown","source":"Another strange thing that I discovered is that there are phrases with a single word only and if they disappear at the following phrases the sentiment changes."},{"metadata":{"trusted":true,"_uuid":"d01853d8d481872f478ebb556507978cd8eb1c81"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 22) & (df['PhraseId'] <= 24)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])\n\nprint()\n\nprint(example[\"Phrase\"].values[2], \" - Sentiment:\", example[\"Sentiment\"].values[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e10221712c1f88a981b35c36f0203d913ffb9d5"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 46) & (df['PhraseId'] <= 47)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa7af87dd828cd396c0289733dc6b74a908d1cf3"},"cell_type":"markdown","source":"As you can see sentence id denotes a single review with the phrase column having the entire review text as an input instance followed by random suffixes of the same sentence to form multiple phrases with subsequent phrase ids. This repeats for every single new sentence id (or new review per se). The sentiment is coded with 5 values 0= Very negative to 4=Very positive and everything else in between.\n\nA quick glance will show you that the data is a little weird for a sentiment corpus:\n\n- Phrases of sentences are** chopped up compeltely randomly**. So logic like sentence tokenization based on periods or punctuations or something of that sort doesn't apply\n- Certain phrases are **with one single word!**.\n- For some phrases inclusion of a punctuation like a comma or a full stop changes the sentiment from say 2 to 3 i.e neutral to positive.\n- Some phrases **starts** with a punctuation like a **backquote**.\n- Some phrases **end** with a **punctuation**\n- There are some ** weird ** words such as ** -RRB-, -LRB- **\n\nAll these weird aspects of this dataset, can be helpful and may be predictive. Afterall, we are looking for patterns in data. Therefore, it would be easier for us to engineer features, I mean apart from the text features that can be extracted from the corpus.\n\nSo, after all this train of thought, let us move on to Machine Learning and Predictive Models."},{"metadata":{"_uuid":"0310a0bc6bac992f98d2cadb99f0d55205187bbc"},"cell_type":"markdown","source":"_________________________"},{"metadata":{"_uuid":"a514bd7e12291708e7594e76b7f0cd0496115bd1"},"cell_type":"markdown","source":"## From Text Data to Feature Extration Representation with TF - IDF using TfidfVectorizer\n\n![TF-IDF](https://i.ytimg.com/vi/bPYJi1E9xeM/maxresdefault.jpg)\n\nTF - IDF is an information retrieval technique that weighs a term’s frequency (TF) and its inverse document frequency (IDF). Each word or term has its respective TF and IDF score. The product of the TF and IDF scores of a term is called the TF * IDF weight of that term. The higher the TF * IDF score (weight), the rarer the term and vice versa. The TF * IDF algorithm is used to weigh a keyword in any content and assign the importance to that keyword based on the number of times it appears in the document. More importantly, it checks how relevant the keyword is throughout the web, which is referred to as corpus [source](https://www.elephate.com/blog/what-is-tf-idf/)."},{"metadata":{"_uuid":"ae8408dd0e3d2b8aa0d77dc8db663a650bd7a3b6"},"cell_type":"markdown","source":"### Creating the TF - IDF vectorizer"},{"metadata":{"trusted":true,"_uuid":"254288ae4e08a3a521a976680ee6a128f04a1dd0"},"cell_type":"code","source":"## TF-IDF and uncleaned dataset\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom nltk.tokenize import TweetTokenizer\n\ntokenizer = TweetTokenizer()\n\nvectorizer = TfidfVectorizer(ngram_range=(1, 3), tokenizer=tokenizer.tokenize)\nfull_text = list(df['Phrase'].values) + list(df_test['Phrase'].values)\n#full_text\n\nvectorizer.fit(full_text)\ntrain_vectorized = vectorizer.transform(df['Phrase'])\ntest_vectorized = vectorizer.transform(df_test['Phrase'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"025c537ad4c98f9335d872e36a76dbefd97a7509"},"cell_type":"markdown","source":"### Shape of TF - IDF vectorizer\nwe can see that due to the fact the ngram_range is from 1 to 3 the columns of the TF - IDF matrix vectorizer is extremely huge. This may lead us to slow down the Machine Learning models to fit the data."},{"metadata":{"trusted":true,"_uuid":"cad9c00fd820c9de1e4ba09ab41a057fe39bd48c"},"cell_type":"code","source":"print(\"Train set dimensions after applying TF-IDF vectorizer:\")\nnp.shape(train_vectorized)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"867154be7375ee47ae0517846c6fe547976f5485"},"cell_type":"markdown","source":"## Machine Learning Techiniques for Multiclass Sentiment Analysis\n\nThe Machine Learning models that will be deployed is the following:\n - **Regression Models**: \n     - LogisticRegression\n - **CART Models**: \n     - DecisionTreeClassifier\n     - ExtraTreeClassifier\n - **Bagging Trees**: \n     - ExtraTreesClassifier\n     - RandomForestClassifier \n - **SVM Models**: \n     - LinearSVC \n - **Naive Bayes Models**: \n     - BernoulliNB\n     - MultinomialNB\n - **Boosting Trees**: \n     - Adaboost Classifier\n     - Extreme Gradient Boosting, XGBoost \n - **Lazy Classifiers**: \n     - KNeighborsClassifier\n \n Since the competion evaluates the models based on accuracy then the models will be evaluated based on accuracy and because the dataset is unbalanced (based on its EDA) us a secondary statistical evaluation metric I will use the F1 score.\n \n The train set will be split in train and validation sets with ratio **80:20** .\n \n For all the ML models the random state will be set to 42 in order to the models be reproducable and create the same results in every run.\n \n Finally as a benchmark model, due to the fact that XGBoost is a state of the art model that is widely used in Machine Learning [source](https://www.kdnuggets.com/2017/10/xgboost-top-machine-learning-method-kaggle-explained.html), it will be used as benchmark and the rest of the Machine Learning models will be compared to its performance"},{"metadata":{"trusted":true,"_uuid":"eeb5184921b7c176006eee073cd05b57c73f03b9"},"cell_type":"code","source":"X = train_vectorized\ny = df.Sentiment.values\n\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import precision_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import classification_report\nimport time\n\n# create a stratified split of the data and a 80/20 split.\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, stratify=y, random_state=42, test_size=0.2, shuffle=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d374e583e465088f8c329bfeeefe91fa6712c544"},"cell_type":"code","source":"ml_default_performance_metrics_df = pd.DataFrame(columns=['accuracy','F1-score','training-time'], index=['LogisticRegression', 'DecisionTreeClassifier', 'ExtraTreeClassifier', 'ExtraTreesClassifier', 'RandomForestClassifier', 'LinearSVC', 'BernoulliNB', 'MultinomialNB', 'AdaboostClassifier', 'XGB', 'KNeighborsClassifier'])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec1f56a7942aa1db6d073cc20610e5cc648d38a6"},"cell_type":"markdown","source":"### Multinomial Logistic Regression\n\n Logistic Regression , the most prevalent algorithm for solving industry scale problems, although its losing ground to other techniques with progress in efficiency and implementation ease of other complex algorithms.\n\nA very convenient and useful side effect of a logistic regression solution is that it doesn’t give you discrete output or outright classes as output. Instead you get probabilities associated with each observation. You can apply many standard and custom performance metrics on this probability score to get a cutoff and in turn classify output in a way which best fits your business problem. Also, logistic regression is pretty efficient in terms of time and memory requirement.\n\nIn addition to above , logistic regression algorithm is robust to small noise in the data and is not particularly affected by mild cases of multi-collinearity. Severe cases of multi-collinearity can be handled by implementing logistic regression with L2 regularization, although if a parsimonious model is needed , L2 regularization is not the best choice because it keeps all the features in the model.\n\nWhere logistic regression starts to falter is, when you have a large number of features and good chunk of missing data. Too many categorical variables are also a problem for logistic regression. Another criticism of logistic regression can be that it uses the entire data for coming up with its scores. Although this is not a problem as such, but it can be argued that “obvious” cases which lie at the extreme end of scores should not really be a concern when you are trying to come up with a separation curve. It should ideally be dependent on those boundary cases, some might argue. Also if some of the features are non-linear, you’ll have to rely on transformations, which become a hassle as size of your feature space increases. We have picked few prominent pros and cons from our discussion to summaries things for logistic regression.\n\nLogistic Regression Pros:\n- Convenient probability scores for observations\n- Efficient implementations available across tools\n- Multi-collinearity is not really an issue and can be countered with L2 regularization to an extent\n- Wide spread industry comfort for logistic regression solutions\nLogistic Regression Cons:\n- Doesn’t perform well when feature space is too large\n- Doesn’t handle large number of categorical features/variables well\n- Relies on transformations for non-linear features\n- Relies on entire data\n\n![logistic-regression](https://www.edvancer.in/wp-content/uploads/2015/10/f5bd5f87059fce20564f6e5eb562022e.png)\n\n[source](https://www.edvancer.in/logistic-regression-vs-decision-trees-vs-svm-part2/)\n"},{"metadata":{"trusted":true,"_uuid":"ca2ffeb62f89fbea5404ec1a528128fc6fcfbc29"},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\n\nstart_time = time.time()\n\nclf_logistic_regression = LogisticRegression(multi_class='ovr', solver='sag', random_state=42)\nclf_logistic_regression.fit(xtrain, ytrain)\npredictions = clf_logistic_regression.predict(xvalid)\n\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\n\nml_default_performance_metrics_df.loc['LogisticRegression']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['LogisticRegression']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['LogisticRegression']['F1-score'] = f1_score(yvalid, predictions, average='micro')\n\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ecfac5065c8f7b29be01a5c3b0097e2d4faf9758"},"cell_type":"markdown","source":"Multinomial Logistic Regression presents a good accuracy 0.63 or 63%. However since the dataset is unbalanced we have to look at the  F1-score and to be more specific the weighted F1-score, We choose the weighted and not the macro or micro because Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall [souce](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html). So the weighted F1 score is 0.59. Furthermore the model's training is extremely fast, it takes only 19 seconds! It has a good precision score in phrases for Sentiment 0 to 4."},{"metadata":{"_uuid":"dcdc4de779bbc3f7228ad6cbc71ae1fff225152f"},"cell_type":"markdown","source":"### DecisionTreeClassifier\n\nDecision trees cut feature space in rectangles which can adjust themselves to any monotonic transformation. Since decision trees are designed to work with discrete intervals or classes of predictors, any number of categorical variables are not really an issue with decision trees. Models obtained from decision tree is fairly intuitive and easy to explain to business. Probability scores are not a direct result but you can use class probabilities assigned to terminal nodes instead. This brings us to the biggest problem associated with Decision Trees, that is, they are highly biased class of models.\n\nDecision Trees Pros:\n- Intuitive Decision Rules\n- Can handle non-linear features\n- Take into account variable interactions\n\nDecision Trees Cons:\n- Highly biased to training set\n- No ranking score as direct result\n\n[source](https://www.edvancer.in/logistic-regression-vs-decision-trees-vs-svm-part2/)\n\n![](https://www.edvancer.in/wp-content/uploads/2015/10/9a299e65388a9609d3d567ee3cf7a6c5.png)"},{"metadata":{"trusted":true,"_uuid":"e32a17d9f1514b411c173995454a44db54f0a904"},"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of DecisionTreeClassifier, with train-test split:\")\n\nclf_DecisionTreeClassifier = DecisionTreeClassifier(random_state=42)\nclf_DecisionTreeClassifier.fit(xtrain, ytrain)\npredictions = clf_DecisionTreeClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\n\nml_default_performance_metrics_df.loc['DecisionTreeClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['DecisionTreeClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['DecisionTreeClassifier']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"94c21f06941bccb5abe80cb46fca7a6b2f4c4d4f"},"cell_type":"markdown","source":"Here with this dataset DecisionTree classifier does not perform very well, its accuracy is only 0.55 and F1-score is 0.53. Based on the classification report the classifier performs well in Neutral class only, it has 0.65 precision on the neutral class. However the classifier missclassifies as Neutral Sentiment most of the other phrases. Hence the F1-score is 0.53."},{"metadata":{"_uuid":"50c945f17e1a55b4bf5530920242dae6b0ca16e2"},"cell_type":"markdown","source":"### ExtraTreeClassifier\n\nExtra-trees differ from classic decision trees in the way they are built. When looking for the best split to separate the samples of a node into two groups, random splits are drawn for each of the max_features randomly selected features and the best split among those is chosen. When max_features is set 1, this amounts to building a totally random decision tree [source](https://scikit-learn.org/stable/modules/generated/sklearn.tree.ExtraTreeRegressor.html)."},{"metadata":{"trusted":true,"_uuid":"81e6e4c3f4812d0f75c6138d56cab7fd07c78d2c"},"cell_type":"code","source":"from sklearn.tree import ExtraTreeClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of ExtraTreeClassifier with train-test split:\")\n\nclf_ExtraTreeClassifier = ExtraTreeClassifier(random_state=42)\nclf_ExtraTreeClassifier.fit(xtrain, ytrain)\npredictions = clf_ExtraTreeClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nml_default_performance_metrics_df.loc['ExtraTreeClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['ExtraTreeClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['ExtraTreeClassifier']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"209370d1e40453a3bd528c9c1837f6de1e7537fe"},"cell_type":"markdown","source":"ExtraTreeClassifier does not perform very well, both accuracy and F1-score are below 0.6 or 60%, 0.58 and 0.57 respectively."},{"metadata":{"_uuid":"f6521221b41fadd05c70db44e3b8ae9ce87e9e8a"},"cell_type":"markdown","source":"### ExtraTreesClassifier\n\nExtraTrees implements a meta estimator that fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting [source](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesRegressor.html)."},{"metadata":{"trusted":true,"_uuid":"2f2d16ff39156326815a8e3c64cbff28f40ca10a"},"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of ExtraTreesClassifier with train-test split:\")\n\nclf_ExtraTreesClassifier = ExtraTreesClassifier(n_estimators=10, random_state=42)\nclf_ExtraTreesClassifier.fit(xtrain, ytrain)\npredictions = clf_ExtraTreesClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['ExtraTreesClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['ExtraTreesClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['ExtraTreesClassifier']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"903fd3239eca85445fccf0e252c059e651f4ecaf"},"cell_type":"markdown","source":"Here we can observe that ExtraTreesClassifier perform quite well, both accuracy and F1-score are above 0.6 or 60%. Especially the classifier has 70% accuracy in neutral phrases."},{"metadata":{"_uuid":"44e92702e4402c6989e27f2fc0ce5e7e8bd17fea"},"cell_type":"markdown","source":"### RandomForestClassifier\n\nRandom forest is just an improvement over the top of the decision tree algorithm. The core idea behind Random Forest is to generate multiple small decision trees from random subsets of the data (hence the name “Random Forest”). Each of the decision tree gives a biased classifier (as it only considers a subset of the data). They each capture different trends in the data. This ensemble of trees is like a team of experts each with a little knowledge over the overall subject but thourough in their area of expertise. Now, in case of classification the majority vote is considered to classify a class. In analogy with experts, it is like asking the same multiple choice question to each expert and taking the answer as the one that most no. of experts vote as correct. In case of Regression, we can use the avg. of all trees as our prediction.In addition to this, we can also weight some more decisive trees high relative to others by testing on the validation data.\n\n[source](https://dimensionless.in/introduction-to-random-forest/)\n\n![](https://dimensionless.in/wp-content/uploads/RandomForest_blog_files/figure-html/voting.png)"},{"metadata":{"trusted":true,"_uuid":"5a9c4b6f0166d40781845a7aa68c50b77f022269"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of RandomForestClassifier with train-test split:\")\n\nclf_RandomForestClassifier = RandomForestClassifier(n_estimators = 10, random_state=42)\nclf_RandomForestClassifier.fit(xtrain, ytrain)\npredictions = clf_RandomForestClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['RandomForestClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['RandomForestClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['RandomForestClassifier']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"905e23ecd2715faa52fefeb007fd409373cc0a27"},"cell_type":"markdown","source":"RandomForest Classifier shows some promising results. Its accuracy and F1-score is 0.62 and 0.60 respectively. However, still the classifier shows great missclassification from phrases with Sentiment 0, 1, 3, 4 to Sentiment 2 based on the high recall to Sentiment 2."},{"metadata":{"_uuid":"1c30dfe3518eec7845dbd522644030299feb321e"},"cell_type":"markdown","source":"### LinearSVC\n\nA Support Vector Machine (SVM) is a supervised machine learning algorithm that can be employed for both classification and regression purposes. SVMs are more commonly used in classification problems. SVMs are based on the idea of finding a hyperplane that best divides a dataset into two classes, as shown in the image below.\n\n![](https://66.media.tumblr.com/ff709fe1c77091952fb3e3e6af91e302/tumblr_inline_o9aa8dYRkB1u37g00_540.png)\n\nSupport vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set.\n\nAs a simple example, for a classification task with only two features (like the image above), you can think of a hyperplane as a line that linearly separates and classifies a set of data.Intuitively, the further from the hyperplane our data points lie, the more confident we are that they have been correctly classified. We therefore want our data points to be as far away from the hyperplane as possible, while still being on the correct side of it. So when new testing data is added, whatever side of the hyperplane it lands will decide the class that we assign to it.\n\nPros:\n- Accuracy\n- Works well on smaller cleaner datasets\n- It can be more efficient because it uses a subset of training points\n\nCons:\n- Isn’t suited to larger datasets as the training time with SVMs can be high\n- Less effective on noisier datasets with overlapping classes\n\n[source](https://www.kdnuggets.com/2016/07/support-vector-machines-simple-explanation.html)."},{"metadata":{"trusted":true,"_uuid":"de643712fcdeff47d49ce6b68765b329847029b6"},"cell_type":"code","source":"from sklearn.svm import LinearSVC\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of LinearSVC, multi_class='ovr', with train-test split:\")\n\nclf_LinearSVC = LinearSVC(multi_class='ovr', random_state=42)\nclf_LinearSVC.fit(xtrain, ytrain)\npredictions = clf_LinearSVC.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['LinearSVC']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['LinearSVC']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['LinearSVC']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fdd0f116a31721af715cbd1e098c1129c6db6b3"},"cell_type":"markdown","source":"LinearSVC classifier exhibits great accuracy so far (0.65) and F1-score 0.64. Nevertheless these encouraging results. The recall in class / Sentiment 2 is high and close to 0.82.\n"},{"metadata":{"_uuid":"a62fe44178f6f74b1824c312e9e4b42e047d28f0"},"cell_type":"markdown","source":"### BernoulliNB, GaussianNB and MultinomialNB\n\nIn machine learning, naive Bayes classifiers are a family of simple \"probabilistic classifiers\" based on applying Bayes' theorem with strong (naive) independence assumptions between the features [source](https://en.wikipedia.org/wiki/Naive_Bayes_classifier).\n\nThe multinomial Naive Bayes classifier is suitable for classification with discrete features (e.g., word counts for text classification). The multinomial distribution normally requires integer feature counts. However, in practice, fractional counts such as tf-idf may also work [source](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html).\n\nLike MultinomialNB, this classifier is suitable for discrete data. The difference is that while MultinomialNB works with occurrence counts, BernoulliNB is designed for binary/boolean features [source](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html)."},{"metadata":{"trusted":true,"_uuid":"783dd7bff29b7bb72face33b69f8cf2b0ca091ea"},"cell_type":"code","source":"from sklearn.naive_bayes import BernoulliNB\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of BernoulliNB with train-test split:\")\n\nclf_BernoulliNB = BernoulliNB()\nclf_BernoulliNB.fit(xtrain, ytrain)\npredictions = clf_BernoulliNB.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['BernoulliNB']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['BernoulliNB']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['BernoulliNB']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"307e770149736f860c67d957289ec0a399b83c74"},"cell_type":"markdown","source":"### MultinomialNB"},{"metadata":{"trusted":true,"_uuid":"8da8dcb5fa2acf88b56a2310ce50bfb4573285c8"},"cell_type":"code","source":"from sklearn.naive_bayes import MultinomialNB\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of MultinomialNB with train-test split:\")\n\nclf_MultinomialNB = MultinomialNB()\nclf_MultinomialNB.fit(xtrain, ytrain)\npredictions = clf_MultinomialNB.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['MultinomialNB']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['MultinomialNB']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['MultinomialNB']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac03a5064f94148a85b760bf3117b5bfc0fb4c63"},"cell_type":"markdown","source":" ### GaussianNB\n \n I can not use GaussianNNB because the TF-IDF train vector is sparse and it can not work with sparse matrices."},{"metadata":{"_uuid":"11fd2d7562e18dd2e7eaa4e59a785b71d17eee62"},"cell_type":"markdown","source":"Both BernouliNB and MultivariateNB classifiers presents not so good accuracy close to 0.6 and F1-score is close to 0.56 and 0.54 respectively. Both have high recall on class 2 which means that they missclassfy lots of the validation data from othejr classes as class 2."},{"metadata":{"_uuid":"c1c691983c031967abd88283e4e0affadf69e71d"},"cell_type":"markdown","source":"### AdaBoostClassifier\n\nThe Adaptive Boosting technique works on improving the areas where the base learner fails. The base learner is a machine learning algorithm which is a weak learner and upon which the boosting method is applied to turn it into a strong learner. Any machine learning algorithm that accept weights on training data can be used as a base learner. In the example taken below, Decision stumps are used as the base learner. The training data are randomly sampled in sample points and from this data they areapplied decision stump algorithm to classify the points. After classifying the sampled points we fit the decision tree stump to the complete training data. This process iteratively happens until the complete training data fits without any error or until a specified maximum number of estimators [source](https://hackernoon.com/boosting-algorithms-adaboost-gradient-boosting-and-xgboost-f74991cad38c)."},{"metadata":{"trusted":true,"_uuid":"e4d05ed347c8107b55bdedfa0fc6121b839bcecc"},"cell_type":"code","source":"from sklearn.ensemble import AdaBoostClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of Adaboost with train-test split:\")\n\nclf_adaboost = AdaBoostClassifier(random_state=42)\nclf_adaboost.fit(xtrain, ytrain)\npredictions = clf_adaboost.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['AdaboostClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['AdaboostClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['AdaboostClassifier']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\n\n### storing performance results:\n\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4a0d5da8e01a41d5c1024d6590352dba8338631b"},"cell_type":"markdown","source":"Unfortunately Adaboost classifier has disappointing results, both accuracy and F1-score are 0.54 and 0.45 respectively."},{"metadata":{"_uuid":"43eb42a9bc71d1933e83a380b5d387ee4388e7fe"},"cell_type":"markdown","source":"### Extreme Gradient Boosting XGBoost\n\nXGBoost is one of the state of the art algorithms. XGBoost is a part of an ensemble of classifiers which are used to win data science competitions. XGBoost is similar to gradient boosting algorithm but it has a few tricks up its sleeve which makes it stand out from the rest.\n\nFeatures of XGBoost are:\n- Clever Penalisation of Trees\n- A Proportional shrinking of leaf nodes\n- Newton Boosting\n- Extra Randomisation Parameter"},{"metadata":{"trusted":true,"_uuid":"960f241ba0d48b581c246c36360ef1a2ee1f2e0d"},"cell_type":"code","source":"import xgboost as xgb\n\nstart_time = time.time()\n\nclf_xgb = xgb.XGBClassifier(objective = 'multi:softmax', seed=42)\nclf_xgb.fit(xtrain, ytrain)\npredictions = clf_xgb.predict(xvalid)\n\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['XGB']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['XGB']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['XGB']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f765e4c3c6bd60dd46004e41461ddc947986d05e"},"cell_type":"markdown","source":"XGBoost classifier performs poorly in this dataset. Only 0.54 accuracy and 0.45 F1-score with default paremeters."},{"metadata":{"_uuid":"497596f8cb3f8755e8eb8c261c475b66d01b7d09"},"cell_type":"markdown","source":"### KNeighborsClassifier\n\nk-nearest neighbors (or k-NN for short) is a simple machine learning algorithm that categorizes an input by using its k nearest neighbors. K-NN is non-parametric, which means that it does not make any assumptions about the probability distribution of the input. This is useful for applications with input properties that are unknown and therefore makes k-NN more robust than algorithms that are parametric. The contrast is that parametric machine learning algorithms tend to produce fewer errors than non-parametric ones, since taking input probabilities into account can influence decision making.\n\nFurthermore, k-NN is a type of lazy learning, which is a learning method that generalizes data in the testing phase, rather than during the training phase. This is contrasted with eager learning, which generalizes data in the training phase rather than the testing phase. A benefit of lazy learning is that it can quickly adapt to changes, since it is not expecting a certain generalized dataset. However, a major downside is that a huge amount of computation occurs during testing (actual use) rather than pre-computation during training [source](https://brilliant.org/wiki/k-nearest-neighbors/)."},{"metadata":{"trusted":true,"_uuid":"b67b4961ffface41f086a7ef3b63241c6cee4d14"},"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier\n\nstart_time = time.time()\n\nclf_knn = KNeighborsClassifier()\nclf_knn.fit(xtrain, ytrain)\npredictions = clf_knn.predict(xvalid)\n\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['KNeighborsClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['KNeighborsClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['KNeighborsClassifier']['F1-score'] = f1_score(yvalid, predictions, average='weighted')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"491d71691b5ca9d8f799db8fa126394ee23de357"},"cell_type":"markdown","source":"Here KNN classifier performs neither well or poorly. Its accuracy and F1-score are both 0.59 however presents in all classes high recall score which means KNN missclassfies most of the cases."},{"metadata":{"_uuid":"908815c4c38638a0a6f0cd808fb899e72b75e1a9"},"cell_type":"markdown","source":"### Summarizing ML Classifiers based on their accuracy with TF - IDF as Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"5e40ffc2120f0f333cc3e5dfcd71f0038d54fe17"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd23e144496ee3decbcc6bee2349ac9835251d8d"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\nml_default_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False).accuracy.plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58dc2688e3b6e48644c034934f73f35c8594f553"},"cell_type":"markdown","source":"### Summarizing ML Classifiers based on their F1-score with TF - IDF as Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"a474e2c628f2a5b26331ed111add021130e85b97"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"F1-score\", ascending=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f2aafd64a393eeb5c966be5cf8f5fc04c077360"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\nml_default_performance_metrics_df.sort_values(by=\"F1-score\", ascending=False)[\"F1-score\"].plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2315b5977428583a0a5b8778d2bf4cb25b57754"},"cell_type":"markdown","source":"### Summarizing ML Classifiers based on their training fitting time with TF - IDF as Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"d2e5935a09fce650a08f08632285182ad5a23ab4"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"training-time\", ascending=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46b19e13668d1e5bba44f496c394037b968f4d81"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\nml_default_performance_metrics_df.sort_values(by=\"training-time\", ascending=True)[\"training-time\"].plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d7a3a40049aaabf3c7496fcba4391d19e8cb5649"},"cell_type":"markdown","source":"### Choosing the Top performed ML classifiers\n\nBased on the bar plots above the top performed ML models based on both accuracy and F1-score that outperform the XGBoost classifier are the following:\n\n- LinearSVC (SVM Classifier)\n- Logistic Regression Classifier\n- ExtraTreesClassifier\n- RandomForestClassfier\n\nThey perform very well in both accuracy and F1-score. So I will move on my ensembling their predictions using the statistical mode to judge what class should be used for my ensemble prediction in my validation set."},{"metadata":{"_uuid":"40bcf2eb8a14b1836b3325845876dcf5864b77a2"},"cell_type":"markdown","source":"### Ensemble Top 4 Models over the validation Set\nThe idea here is to ensemble ML models' predictions using the statistical mode over the predicted classes."},{"metadata":{"trusted":true,"_uuid":"f71b70765f5470eac5a53ee2866a095148d518b4"},"cell_type":"code","source":"predictions_linear_svc = clf_LinearSVC.predict(xvalid)\npredictions_extra_trees = clf_ExtraTreesClassifier.predict(xvalid)\npredictions_logistic_regression = clf_logistic_regression.predict(xvalid)\npredictions_random_forest = clf_RandomForestClassifier.predict(xvalid)\n\nensemble_all_ml_default_pred_df = pd.DataFrame({\n    'linear_svc':predictions_linear_svc,\n    'logistic_regression':predictions_logistic_regression,\n    'extra_trees':predictions_extra_trees,\n    'random_forest':predictions_random_forest})\n\n\n# ensemble classes' predictions over the validation set\npred_mode = ensemble_all_ml_default_pred_df.agg('mode',axis=1)[0].values\n\nprint(classification_report(yvalid, pred_mode))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, pred_mode))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, pred_mode, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, pred_mode, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, pred_mode, average='weighted'))\n\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, pred_mode)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d27c016e1e0bb00ec855751b60be9a283a1a170"},"cell_type":"markdown","source":"### Producing Predictions for the Test Set for the top 4 best performed ML models with default paremeters\nWe are know getting prepared to create submission csv files for Kaggle's Movie Review competion to submit them and measure out ML models performance over the Test Set."},{"metadata":{"trusted":true,"_uuid":"85f66b8d77257a9e3a7cd2380e1241744cc4006e"},"cell_type":"code","source":"df_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")\n\ntest_vectorized = vectorizer.transform(df_test['Phrase'])\n\npredictions_default_linear_svc = clf_LinearSVC.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_linear_svc\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_linear_svc.csv',index=False)\n\npredictions_default_logistic_regression = clf_logistic_regression.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_logistic_regression\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_logistic_regression.csv',index=False)\n\npredictions_default_extra_trees = clf_ExtraTreesClassifier.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_extra_trees\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_extra_trees.csv',index=False)\n\npredictions_default_random_forest = clf_RandomForestClassifier.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_random_forest\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_random_forest.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1ddad3607b6343f93b6f67cb84db318a348fffa4"},"cell_type":"markdown","source":"### Ensemble Predictions from Test Set from the top 5 best performed ML models with default parameters\nSo we do and for out top 4 ML models ensemble predictions over the Test Set using the statistical mode as ensemble criterion and preparing a csv submission file to measure our accuracy over the Test Set."},{"metadata":{"trusted":true,"_uuid":"3b0e763dc62a105d7c0949f1d17c5e9376117ff7"},"cell_type":"code","source":"ensemble_all_ml_default_pred_test_df = pd.DataFrame({\n    'linear_svc':predictions_default_linear_svc,\n    'logistic_regression':predictions_default_logistic_regression,\n    'extra_trees':predictions_default_extra_trees,\n    'random_forest':predictions_default_random_forest})\n\n# ensemble classes' predictions over the validation set\npred_default_test_mode = ensemble_all_ml_default_pred_test_df.agg('mode',axis=1)[0].values\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = pred_default_test_mode\nsubmission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_ml_default_ensemble.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2400e8434e3ce07aaaec416dc7bbe862f0664d9"},"cell_type":"markdown","source":"### Tuning the top 4 best performed ML Models\n\nSince the top 4 Machine Learning models performed really well and evaluated in both accuracy and F1-score metric using their defauly paremeters as they are, the next step is to tune them in the same train - test split and see if they perform even better. To sum up the ML models that will be tuned are the following:\n\n- SVM\n- Logistic Regression\n- ExtraTrees\n- RandomForest"},{"metadata":{"trusted":true,"_uuid":"629882a76687533788b8328a85432b0a66df38c7"},"cell_type":"code","source":"ml_tuning_performance_metrics_df = pd.DataFrame(columns=['accuracy','F1-score','training-time'], index=['LogisticRegression', 'ExtraTreesClassifier', 'RandomForestClassifier', 'LinearSVC'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b761421e07eb96d79c04098267cf1f0aff646388"},"cell_type":"markdown","source":"### LinearSVC tuning"},{"metadata":{"_uuid":"21af35e66c02922cc0da6c5dfabfda0d786b5231","trusted":true},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nfrom sklearn.svm import LinearSVC\nimport time\n\nstart_time = time.time()\nprint(\"Linear SVC grid Search:\")\n\nclf_linear_svc_grid = LinearSVC(multi_class='ovr', random_state=42)\n\nX = train_vectorized\ny = df.Sentiment.values\n\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, stratify=y, random_state=42, test_size=0.2, shuffle=True)\n\ngsc_linear_svc = GridSearchCV(\n    estimator=clf_linear_svc_grid,\n    param_grid={\n        'C': [10e-1, 10e-2, 1, 10],\n        'tol': [10e-5, 10e-4, 10e-3, 10e-2, 10e-1, 10e-0, 10e+1],\n    },\n    scoring='accuracy',\n    cv=2\n)\n\ngrid_result_linear_svc = gsc_linear_svc.fit(xtrain, ytrain)\n\nprint()\nprint(\"ExtraTreesClassifier best parameters and score\")\nprint(\"Best score: %f using %s\" % (grid_result_linear_svc.best_score_, grid_result_linear_svc.best_params_))\n\nprediction_tuned_linear_svc = gsc_linear_svc.predict(xvalid)\nprint(\"accuracy score after tuning:\", accuracy_score(yvalid, prediction_tuned_linear_svc))\n\nml_tuning_performance_metrics_df.loc['LinearSVC']['training-time'] = time.time() - start_time\nml_tuning_performance_metrics_df.loc['LinearSVC']['accuracy'] = accuracy_score(yvalid, prediction_tuned_linear_svc)\nml_tuning_performance_metrics_df.loc['LinearSVC']['F1-score'] = f1_score(yvalid, prediction_tuned_linear_svc, average='macro')\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"600d4376fbe5358fe2011f11bbddc5c2240c08eb"},"cell_type":"markdown","source":"### LinearSVC tuning evaluation"},{"metadata":{"trusted":true,"_uuid":"0f6a54312775a648e45c9760551af09eb6fff85a"},"cell_type":"code","source":"predictions_tuned_linear_svc = grid_result_linear_svc.predict(xvalid)\n\nprint(classification_report(yvalid, predictions_tuned_linear_svc))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions_tuned_linear_svc))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions_tuned_linear_svc, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions_tuned_linear_svc, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions_tuned_linear_svc, average='weighted'))\n\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions_tuned_linear_svc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1cf28f95ec27a8696b29218a5a80c94300fd22f6"},"cell_type":"markdown","source":"Although the LinearSVC tuning, the accuracy and the F1-score did not improve as much as we wanted to. Moreover, the ML model suffers from high recall."},{"metadata":{"_uuid":"474fd780f1a31e31bde1bc72fa550bfbb820b04a"},"cell_type":"markdown","source":"### Logistic Regression Tuning"},{"metadata":{"trusted":true,"_uuid":"8eb44adf26580c0daee090cd8bc53d46cfb2674c"},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nfrom sklearn.linear_model import LogisticRegression\nimport time\n\nstart_time = time.time()\nprint(\"LogisticRegression grid Search:\")\n\nclf_logistic_regression_grid = LogisticRegression(multi_class = 'ovr', random_state=42)\n\nX = train_vectorized\ny = df.Sentiment.values\n\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, stratify=y, random_state=42, test_size=0.2, shuffle=True)\n\ngsc_logistic_regression = GridSearchCV(\n    estimator=clf_logistic_regression_grid,\n    param_grid={\n        'solver': ['newton-cg', 'sag', 'saga'],\n        'C': [10e-1, 10e-2, 1, 10],\n        'tol': [10e-4, 10e-3, 10e-2, 10e-1, 10e-0],\n    },\n    scoring='accuracy',\n    cv=2\n)\n\ngrid_result_logistic_regression = gsc_logistic_regression.fit(xtrain, ytrain)\n\nprint()\nprint(\"LogisticRegression best parameters and score\")\nprint(\"Best score: %f using %s\" % (grid_result_logistic_regression.best_score_, grid_result_logistic_regression.best_params_))\n\nprediction_tuned_logistic_regression = grid_result_logistic_regression.predict(xvalid)\nprint(\"accuracy score after tuning:\", accuracy_score(yvalid, prediction_tuned_logistic_regression))\n\nml_tuning_performance_metrics_df.loc['LogisticRegression']['training-time'] = time.time() - start_time\nml_tuning_performance_metrics_df.loc['LogisticRegression']['accuracy'] = accuracy_score(yvalid, prediction_tuned_logistic_regression)\nml_tuning_performance_metrics_df.loc['LogisticRegression']['F1-score'] = f1_score(yvalid, prediction_tuned_logistic_regression, average='macro')\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc58c4392fcd230aaa3bd9c87ee8e46df12c46d4"},"cell_type":"markdown","source":"### LogisticRegression tuning evaluation"},{"metadata":{"trusted":true,"_uuid":"564a992ceacf201f851696961f8ca3f9f235c243"},"cell_type":"code","source":"predictions_tuned_logistic_regression = grid_result_logistic_regression.predict(xvalid)\n\nprint(classification_report(yvalid, predictions_tuned_logistic_regression))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions_tuned_logistic_regression))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions_tuned_logistic_regression, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions_tuned_logistic_regression, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions_tuned_logistic_regression, average='weighted'))\n\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions_tuned_logistic_regression)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd3a7c4ddc72eb85233690d48c280323275dad8c"},"cell_type":"markdown","source":"Unfortunately after tuning still LogisticRegression classifier still can't classifier with great accuracy and F1-score the majority of the cases. In fact it missclassifies most of them and this can be seen by the high recall from class to class."},{"metadata":{"_uuid":"eafb178a9f84d6ed50f7c532e398edbc813367e8"},"cell_type":"markdown","source":"### ExtraTrees Tuning"},{"metadata":{"trusted":true,"_uuid":"aeb18b63b80459df9086acebd608d6ea884801ff"},"cell_type":"code","source":"# I will perform a shallow grid search \n\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.ensemble import ExtraTreesClassifier\nimport time\n\nstart_time = time.time()\nprint(\"ExtraTreesClassifier grid Search:\")\n\nclf_extra_trees_grid = ExtraTreesClassifier(random_state=42)\n\nX = train_vectorized\ny = df.Sentiment.values\n\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, stratify=y, random_state=42, test_size=0.2, shuffle=True)\n\ngsc_extra_trees = GridSearchCV(\n    estimator=clf_extra_trees_grid,\n    param_grid={\n        'n_estimators': [2, 4, 6, 8, 10, 12]\n    },\n    scoring='accuracy',\n    cv=2\n)\n\ngrid_result_extra_trees = gsc_extra_trees.fit(xtrain, ytrain)\n\nprint()\nprint(\"ExtraTreesClassifier best parameters and score\")\nprint(\"Best score: %f using %s\" % (grid_result_extra_trees.best_score_, grid_result_extra_trees.best_params_))\n\nprediction_tuned_extra_trees = grid_result_extra_trees.predict(xvalid)\nprint(\"accuracy score after tuning:\", accuracy_score(yvalid, prediction_tuned_extra_trees))\n\nml_tuning_performance_metrics_df.loc['ExtraTreesClassifier']['training-time'] = time.time() - start_time\nml_tuning_performance_metrics_df.loc['ExtraTreesClassifier']['accuracy'] = accuracy_score(yvalid, prediction_tuned_extra_trees)\nml_tuning_performance_metrics_df.loc['ExtraTreesClassifier']['F1-score'] = f1_score(yvalid, prediction_tuned_extra_trees, average='macro')\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8aa1448f4426a29241f385cd27a2d1a0e51c4e1b"},"cell_type":"markdown","source":"### ExtraTrees tuning evaluation"},{"metadata":{"trusted":true,"_uuid":"d30e454c8146133184929df3abafd30ef0b8870c"},"cell_type":"code","source":"predictions_tuned_extra_trees = grid_result_extra_trees.predict(xvalid)\n\nprint(classification_report(yvalid, predictions_tuned_extra_trees))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions_tuned_extra_trees))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions_tuned_extra_trees, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions_tuned_extra_trees, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions_tuned_extra_trees, average='weighted'))\n\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions_tuned_extra_trees)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2021c7f991494d463f0eb1543bca5ad8e069dd4"},"cell_type":"markdown","source":"The same situation and with ExtraTrees tuning, the accuracy and the F1-score did not improve as much as we wanted to. Furthermore, even now the ML model suffers from high recall."},{"metadata":{"_uuid":"4db8c4544ef1375244a626a057c51ad58d5b3604"},"cell_type":"markdown","source":"### RandomForestClassifier Tuning"},{"metadata":{"trusted":true,"_uuid":"be24c932ba2406575c0aca38cab283dad4725543"},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nfrom sklearn.model_selection import StratifiedShuffleSplit\nfrom sklearn.ensemble import RandomForestClassifier\n\nstart_time = time.time()\nprint(\"RandomForestClassifier grid Search:\")\n\nclf_random_forest = RandomForestClassifier(random_state=42)\n\nX = train_vectorized\ny = df.Sentiment.values\n\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, stratify=y, random_state=42, test_size=0.2, shuffle=True)\n\ngsc_random_forest = GridSearchCV(\n    estimator=clf_random_forest,\n    param_grid={\n        'n_estimators': [2, 4, 6, 8, 10, 12]\n    },\n    scoring='accuracy',\n    cv=2\n)\n\ngrid_result_random_forest = gsc_random_forest.fit(xtrain, ytrain)\n\nprint()\nprint(\"RandomForest best parameters and score\")\nprint(\"Best: %f using %s\" % (grid_result_random_forest.best_score_, grid_result_random_forest.best_params_))\n\nprediction_tuned_random_forest = grid_result_random_forest.predict(xvalid)\nprint(\"accuracy score after tuning:\", accuracy_score(yvalid, prediction_tuned_random_forest))\n\nml_tuning_performance_metrics_df.loc['RandomForestClassifier']['training-time'] = time.time() - start_time\nml_tuning_performance_metrics_df.loc['RandomForestClassifier']['accuracy'] = accuracy_score(yvalid, prediction_tuned_random_forest)\nml_tuning_performance_metrics_df.loc['RandomForestClassifier']['F1-score'] = f1_score(yvalid, prediction_tuned_random_forest, average='macro')\n\nprint()\nprint(\"elapsed time in seconds: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d70b98a13100fa6e2a094f38e5652f75f6b18599"},"cell_type":"markdown","source":"### RandomForest tuning evaluation"},{"metadata":{"trusted":true,"_uuid":"a7c96a34970ed7b590b1269878e5ff189a77447f"},"cell_type":"code","source":"predictions_tuned_random_forest = grid_result_random_forest.predict(xvalid)\n\nprint(classification_report(yvalid, predictions_tuned_random_forest))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions_tuned_random_forest))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions_tuned_random_forest, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions_tuned_random_forest, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions_tuned_random_forest, average='weighted'))\n\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions_tuned_random_forest)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0b45a0f320025b730d111713d22339f0443379f"},"cell_type":"markdown","source":"Again the RandomForest tuning, the accuracy and the F1-score did not improve as much as we wanted to. Still the ML model continues to missclasify most of the cases as of a class #2."},{"metadata":{"_uuid":"224f1418ac15741dc04d176a8ba7d78fba860c2d"},"cell_type":"markdown","source":"### Comparing accuracy and F1-score between default and tuned parameters in ML models"},{"metadata":{"trusted":true,"_uuid":"e77459b566097a72b9822909540d13b48762e981"},"cell_type":"code","source":"ml_tuning_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e6d44a02f1fc7ad66c33f902993b074b61cd9cb"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"F1-score\", ascending=False).head(4)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3be70f476be3b196c7df1b60d2af959d809d4b6a"},"cell_type":"markdown","source":"Although some improvements in accuracy  and F1-score tuning in fact provided some minor advantages to out models based on the two tables above."},{"metadata":{"_uuid":"312d8ff4f6a4ac28d4de1e47bd895168b7f52f71"},"cell_type":"markdown","source":"### Ensemble The top 4 Tuned Models over the validation Set\nLet us ensemble now the tune ML models over the validation set using the statistical mode over the ML models' multiclass predictions and evaluate its performance."},{"metadata":{"trusted":true,"_uuid":"5b34a778c1aa71a1b3073869f48ffc82713eb699"},"cell_type":"code","source":"predictions_tuned_linear_svc = grid_result_linear_svc.predict(xvalid)\npredictions_tuned_logistic_regression = grid_result_logistic_regression.predict(xvalid)\npredictions_tuned_extra_trees = grid_result_extra_trees.predict(xvalid)\npredictions_tuned_random_forest = grid_result_random_forest.predict(xvalid)\n\nensemble_all_ml_tuned_pred_df = pd.DataFrame({\n    'linear_svc':predictions_tuned_linear_svc,\n    'logistic_regression':predictions_tuned_logistic_regression,\n    'extra_trees':predictions_tuned_extra_trees,\n    'random_forest':predictions_tuned_random_forest})\n\npred_tuned_mode = ensemble_all_ml_tuned_pred_df.agg('mode',axis=1)[0].values\n\nprint(classification_report(yvalid, pred_tuned_mode))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, pred_tuned_mode))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, pred_tuned_mode, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, pred_tuned_mode, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, pred_tuned_mode, average='weighted'))\n\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, pred_tuned_mode)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b980110abd1133253ea68a54d84e21927b51e38"},"cell_type":"markdown","source":"Even ensembling the tuned predictions over the validation set using the statistical mode, we can not overcome and improve the accuracy over 0.65 and the same happens to F1-score. Still many cases are missclassified as class #2."},{"metadata":{"_uuid":"061c8b376670d17752d0954c40ff6ff96680104c"},"cell_type":"markdown","source":"### ML Predictions with Test Set\nLets prepare a sumbission for Kaggle's Competition"},{"metadata":{"trusted":true,"_uuid":"963b1df7f352ae720e83eb228f0bce16588fcff2"},"cell_type":"code","source":"df_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")\n\ntest_vectorized = vectorizer.transform(df_test['Phrase'])\n\npredictions_tuned_test_linear_svc = grid_result_linear_svc.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_tuned_test_linear_svc\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_tuned_linear_svc.csv',index=False)\n\npredictions_tuned_test_logistic_regression = grid_result_logistic_regression.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_tuned_test_logistic_regression\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_tuned_logistic_regression.csv',index=False)\n\npredictions_tuned_test_extra_trees = grid_result_extra_trees.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_tuned_test_extra_trees\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_tuned_extra_trees.csv',index=False)\n\npredictions_tuned_test_random_forest = grid_result_random_forest.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_tuned_test_random_forest\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_tuned_random_forest.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74df6b8459735c4676b2a206d689d3c0905eb289"},"cell_type":"markdown","source":"### Ensemble Predictions after Tuning from top 4 best performed ML Techniques after tuning\nEnsemble submission for Kaggle Competition"},{"metadata":{"trusted":true,"_uuid":"8ae6eada301e72aec2503407eb7c596baf2e1dc3"},"cell_type":"code","source":"ensemble_all_ml_tuned_pred_test_df = pd.DataFrame({\n    'linear_svc':predictions_tuned_test_linear_svc,\n    'logistic_regression':predictions_tuned_test_logistic_regression,\n    'extra_trees':predictions_tuned_test_extra_trees,\n    'random_forest':predictions_tuned_test_random_forest})\n\n\npred_tuned_test_mode = ensemble_all_ml_tuned_pred_test_df.agg('mode',axis=1)[0].values\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = pred_tuned_test_mode\nsubmission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_ml_tuned_ensemble.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2210183c0c172aed22025c0fd0136310aa3c2f3c"},"cell_type":"markdown","source":"## Summary of Machine Learning and TF - IDF feature extraction / representation\nIin general most of the ML models tried to fit on this strange dataset with phrases even with one word and the absence of a single punctuation symbol can change the phrase's sentiment. Nevertheless LogisticRegression, ExtraTrees, LinearSVC and RandomForest presented good results and outperformed the famous XGBoost classifier but with high recalls especially in class #2 the \"Neutral\" sentiment. Tuning showed minor improvements. The next phase is to use word embeddings combined with Machine Learning models and see if there are some better improvements there."},{"metadata":{"_uuid":"a74b4df01c678a89730431bfe3a24a2b1793f822"},"cell_type":"markdown","source":"___________________________________________________________________"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}