{"cells":[{"metadata":{"_uuid":"d088ad8cad761921f1b9040eb9322b62fd799e5a"},"cell_type":"markdown","source":"p# NLP Capstone Project - Movie Review - Sentiment Analysis -  Classify the sentiment of sentences from the Rotten Tomatoes dataset\n## Developing Machine Learning Models using pre-trained GLoVe Word embeddings as feature extraction / representation\n\n![](https://cdn.steemitimages.com/DQmQZCf7ME7Haj3X3MzXtG8R8JtGmTpuh5NXDSd3wKueva7/rottentomatoes.png)\n\n![Sentiment](https://www.kdnuggets.com/images/sentiment-fig-1-689.jpg)\n\n![ML](https://juststickers.in/wp-content/uploads/2017/04/machine-learning.png)\n\n![W2v](https://newvitruvian.com/images/term-vector-word2vec-4.png)\n\nMoving one from EDA and Machine Learning models with TF - IDF Feature EXtraction / Representation, another feature representation technique is to convert the words from inside the phrases from the movies reviews into dense vectors. In particular the ML models will be developed in conjunction with Word Embeddings as feature representation.\n\nAs Word Embeddings already pre-trained Word Embeddings will be used from [Stanford NLP](https://nlp.stanford.edu/projects/glove/).\n\nAt first, There must be mention again that after EDA and ML models with TF / IDF again an odd conclusion was made. The dataset of this competition turned to have some unique features. we have only phrases as data. And a phrase can contain a single word. And one punctuation mark can cause phrase to receive a different sentiment. Also assigned sentiments can be strange. This means several things:\n\n- using stopwords can be a bad idea, especially when phrases contain one single stopword;\n- puntuation could be important, so it should be used;\n\n** This thought will be enhanced later with my anomaly detection insights **"},{"metadata":{"_uuid":"f5c933949f6eef7669ee8f554cab65bbcf456c15"},"cell_type":"markdown","source":"## Loading Main Libraries"},{"metadata":{"trusted":true,"_uuid":"279572634fb1985999424d259fc038eb12732c9a"},"cell_type":"code","source":"import pandas as pd\n\nimport matplotlib\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport nltk\n\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1079c746f344c1d127d797215681eee1c3727739"},"cell_type":"markdown","source":"### load the dataset"},{"metadata":{"trusted":true,"_uuid":"8ed18e14e5d260be8729fb4a815f570b66ed1c72"},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.tsv\", sep=\"\\t\")\n\ndf_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"509ecba55df17d0dc7f797aa2df25fb357f81bed"},"cell_type":"markdown","source":"### Preview"},{"metadata":{"trusted":true,"_uuid":"f1bc83fe85a53c3aab52fa91964cc9d2a7dcd5c3"},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e377c1e3d5a11a219c0ebdb7f4b852459f1653b0"},"cell_type":"code","source":"df_test.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58b5e30a37d49cbe966c0721fa7319774c691bba"},"cell_type":"markdown","source":"### Thoughts before training ML models\nHere are couple of instances where punctuations appeared to be predictive. So if we \"cleanedup\" the data in the name of data preparation some predictiveness will be lost."},{"metadata":{"trusted":true,"_uuid":"1444073d8d53f00c36d5171ca929f3964850a94f"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 0) & (df['PhraseId'] <= 2)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6eaf24187402262a83b60e419ddd9bbd67394513"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 517) & (df['PhraseId'] <= 518)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08f508da8d6535ff9362bcc3edbc162e77080bbc"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \",\" is important"},{"metadata":{"trusted":true,"_uuid":"844d5f7d4c0528fc883f0b28dcf86f5665d2341d"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 68) & (df['PhraseId'] <= 69)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_uuid":"35a464345540a822ea814b06ddb95596cfd2fb67"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \"!\" is important"},{"metadata":{"trusted":true,"_uuid":"abfde8aea697aa803e36b0372d6a76912cc66e00"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 10737) & (df['PhraseId'] <= 10738)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"072f7b4a74b61b6b7c08a42b21694d46ea315c76"},"cell_type":"markdown","source":"Another strange thing that I discovered is that there are phrases with a single word only and if they disappear at the following phrases the sentiment changes."},{"metadata":{"trusted":true,"_uuid":"94512a174921bd3b63fe4b67a8ab84718689b267"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 22) & (df['PhraseId'] <= 24)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])\n\nprint()\n\nprint(example[\"Phrase\"].values[2], \" - Sentiment:\", example[\"Sentiment\"].values[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3e6ef6f4dc3739122f14309b538b514315ab44b"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 46) & (df['PhraseId'] <= 47)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"652c0cb0e9e22d88b0302d9d2a1b91141b196434"},"cell_type":"markdown","source":"As you can see sentence id denotes a single review with the phrase column having the entire review text as an input instance followed by random suffixes of the same sentence to form multiple phrases with subsequent phrase ids. This repeats for every single new sentence id (or new review per se). The sentiment is coded with 5 values 0= Very negative to 4=Very positive and everything else in between.\n\nA quick glance will show you that the data is a little weird for a sentiment corpus:\n\n- Phrases of sentences are** chopped up compeltely randomly**. So logic like sentence tokenization based on periods or punctuations or something of that sort doesn't apply\n- Certain phrases are **with one single word!**.\n- For some phrases inclusion of a punctuation like a comma or a full stop changes the sentiment from say 2 to 3 i.e neutral to positive.\n- Some phrases **starts** with a punctuation like a **backquote**.\n- Some phrases **end** with a **punctuation**\n- There are some ** weird ** words such as ** -RRB-,  -LRB- **\n\nAll these weird aspects of this dataset, can be helpful and may be predictive. Afterall, we are looking for patterns in data. Therefore, it would be easier for us to engineer features, I mean apart from the text features that can be extracted from the corpus.\n\nSo, after all this train of thought, let us move on to Machine Learning and Predictive Models."},{"metadata":{"_uuid":"0310a0bc6bac992f98d2cadb99f0d55205187bbc"},"cell_type":"markdown","source":"_________________________"},{"metadata":{"_uuid":"cecbfdc31929f5dc114769154a21a5604876d7bb"},"cell_type":"markdown","source":"## Word to feature Extraction with pre-trained Word Embeddings\n![w2v2](https://cdn-images-1.medium.com/max/1600/1*jpnKO5X0Ii8PVdQYFO2z1Q.png)\n\nWord embedding is one of the most popular representation of document vocabulary. It is capable of capturing context of a word in a document, semantic and syntactic similarity, relation with other words, etc.\n\nWord Embeddings originates from the idea of generating distributed representations. Intuitively, there is some dependence of one word on the other words. The words in context of this word would get a greater share of this dependence. In one hot encoding representations, all the words are independent of each other, as mentioned earlier.\n\nWord Embeddings are vector representations of a particular word. Word Embeddings is a method to construct such an embedding. It can be obtained using two methods (both involving Neural Networks): Skip Gram and Common Bag Of Words. [source](https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa).\n\nTo get Word Embeddings they will be downloaded from Stanford NLP GloVe: **Global Vectors for Word Representation**. GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space [source](https://nlp.stanford.edu/projects/glove/).\n\nSo we continue with the download of the GLoVe embeddings, The embedding size of 300 will be used for increased expressibility."},{"metadata":{"trusted":true,"_uuid":"6af47865bfba9203ef0999c1afe1a80f800b0b5b"},"cell_type":"code","source":"#download GLoVe Embeddings\n\nimport os,requests\n\ndef download(url):\n    get_response = requests.get(url,stream=True)\n    file_name  = url.split(\"/\")[-1]\n    with open(file_name, 'wb') as f:\n        for chunk in get_response.iter_content(chunk_size=1024):\n            if chunk: # filter out keep-alive new chunks\n                f.write(chunk)\n        \n\ndownload(\"http://nlp.stanford.edu/data/glove.6B.zip\")\n        \n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"32bdcf91d965e2d5c4d63a2cd4adf510e0d53231"},"cell_type":"markdown","source":"### Export the GLoVe zip file."},{"metadata":{"trusted":true,"_uuid":"2e4c7da2c30ae67357a9dec2d770064dad8dcfa2"},"cell_type":"code","source":"# Export the GLoVe zip file.\n\nimport zipfile\n\nwith zipfile.ZipFile('glove.6B.zip', 'r') as zip_ref:\n    zip_ref.extractall('.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2edd02b8af17155843c6b2d384d135b398327e27"},"cell_type":"code","source":"os.listdir()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67615bbdc49e97599bc17ea9e2797738d59ac3a5"},"cell_type":"markdown","source":"### Reading the GLoVe embedding txt file"},{"metadata":{"trusted":true,"_uuid":"bcf5e33d89b9d3c93ad0478851172d7cb7f23369"},"cell_type":"code","source":"embedding_dim = 300\nfilename = 'glove.6B.'+ str(embedding_dim) +'d.txt'\n\nglove_w2v_embeddings_index = dict()\nf = open(filename, \"r\", encoding='utf-8')\nfor line in f:\n    values = line.split()\n    word = values[0]\n    coefs = np.asarray(values[1:], dtype='float32')\n    glove_w2v_embeddings_index[word] = coefs\nf.close()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cdf5d207b1969329e466e8efa31f0e4cdea327a6"},"cell_type":"markdown","source":"### Averaging glove embeddings\nSince each word is a vector with size of 300. We have phrases that have different number of words. To solve this issue all the word vectors per phrase will be averaged to reduce a phrase with different words to a averaged word vector with dimension of 300."},{"metadata":{"trusted":true,"_uuid":"1d5e00a1f814eadf23d6ec580cf9520dd1ee7ae8"},"cell_type":"code","source":"sentences = df['Phrase'].values\nmajor_sent = []\n\nfor sent in sentences:\n    \n    temp_sent = []\n    \n    for word in sent.split(\" \"):    \n        \n        if word in glove_w2v_embeddings_index:\n            temp = glove_w2v_embeddings_index['word']\n        else:\n            temp = np.zeros(embedding_dim)\n\n        temp_sent.append(temp)\n    \n    temp_sent = np.mean(temp_sent, axis=0)\n    major_sent.append(temp_sent)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42c9ce75ee00af5a67964deb9ec00245ae5af5a9"},"cell_type":"code","source":"print(\"Train Set dimensions after averaging Word Embeddings:\")\nprint(np.shape(major_sent))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"867154be7375ee47ae0517846c6fe547976f5485"},"cell_type":"markdown","source":"## Machine Learning Techiniques for Multiclass Sentiment Analysis\n\nThe Machine Learning models that will be deployed is the following:\n - **Regression Models**: \n     - LogisticRegression\n - **CART Models**: \n     - DecisionTreeClassifier\n     - ExtraTreeClassifier\n - **Bagging Trees**: \n     - ExtraTreesClassifier\n     - RandomForestClassifier \n - **SVM Models**: \n     - LinearSVC \n - **Naive Bayes Models**: \n     - BernoulliNB\n     - MultinomialNB\n - **Boosting Trees**: \n     - Adaboost Classifier\n     - Extreme Gradient Boosting, XGBoost \n - **Lazy Classifiers**: \n     - KNeighborsClassifier\n \n Since the competion evaluates the models based on accuracy then the models will be evaluated based on accuracy and because the dataset is unbalanced (based on its EDA) us a secondary statistical evaluation metric I will use the F1 score.\n \n The train set will be split in train and validation sets with ratio **80:20** .\n \n For all the ML models the random state will be set to 42 in order to the models be reproducable and create the same results in every run.\n \n Finally as a benchmark model, due to the fact that XGBoost is a state of the art model that is widely used in Machine Learning [source](https://www.kdnuggets.com/2017/10/xgboost-top-machine-learning-method-kaggle-explained.html), it will be used as benchmark and the rest of the Machine Learning models will be compared to its performance"},{"metadata":{"trusted":true,"_uuid":"eeb5184921b7c176006eee073cd05b57c73f03b9"},"cell_type":"code","source":"X = major_sent\ny = df.Sentiment.values\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import precision_score\nfrom sklearn.metrics import recall_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import classification_report\nimport time\n\nxtrain, xvalid, ytrain, yvalid = train_test_split(X, y, random_state=42, test_size=0.2, shuffle=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d374e583e465088f8c329bfeeefe91fa6712c544"},"cell_type":"code","source":"ml_default_performance_metrics_df = pd.DataFrame(columns=['accuracy','F1-score','training-time'], index=['LogisticRegression', 'DecisionTreeClassifier', 'ExtraTreeClassifier', 'ExtraTreesClassifier', 'RandomForestClassifier', 'LinearSVC', 'BernoulliNB', 'AdaboostClassifier', 'XGB', 'KNeighborsClassifier'])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec1f56a7942aa1db6d073cc20610e5cc648d38a6"},"cell_type":"markdown","source":"### Multinomial Logistic Regression"},{"metadata":{"trusted":true,"_uuid":"ca2ffeb62f89fbea5404ec1a528128fc6fcfbc29"},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\n\nimport sys\nimport warnings\n\nif not sys.warnoptions:\n    warnings.simplefilter(\"ignore\")\n\nstart_time = time.time()\n\nclf_logistic_regression = LogisticRegression(multi_class='ovr', solver='sag', random_state=42)\nclf_logistic_regression.fit(xtrain, ytrain)\npredictions = clf_logistic_regression.predict(xvalid)\n\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\n\nml_default_performance_metrics_df.loc['LogisticRegression']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['LogisticRegression']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['LogisticRegression']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcdc4de779bbc3f7228ad6cbc71ae1fff225152f"},"cell_type":"markdown","source":"### DecisionTreeClassifier"},{"metadata":{"trusted":true,"_uuid":"e32a17d9f1514b411c173995454a44db54f0a904"},"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of DecisionTreeClassifier, with train-test split:\")\n\nclf_DecisionTreeClassifier = DecisionTreeClassifier(random_state=42)\nclf_DecisionTreeClassifier.fit(xtrain, ytrain)\npredictions = clf_DecisionTreeClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\n\nml_default_performance_metrics_df.loc['DecisionTreeClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['DecisionTreeClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['DecisionTreeClassifier']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50c945f17e1a55b4bf5530920242dae6b0ca16e2"},"cell_type":"markdown","source":"### ExtraTreeClassifier"},{"metadata":{"trusted":true,"_uuid":"81e6e4c3f4812d0f75c6138d56cab7fd07c78d2c"},"cell_type":"code","source":"from sklearn.tree import ExtraTreeClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of ExtraTreeClassifier with train-test split:\")\n\nclf_ExtraTreeClassifier = ExtraTreeClassifier()\nclf_ExtraTreeClassifier.fit(xtrain, ytrain)\npredictions = clf_ExtraTreeClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nml_default_performance_metrics_df.loc['ExtraTreeClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['ExtraTreeClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['ExtraTreeClassifier']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6521221b41fadd05c70db44e3b8ae9ce87e9e8a"},"cell_type":"markdown","source":"### ExtraTreesClassifier"},{"metadata":{"trusted":true,"_uuid":"2f2d16ff39156326815a8e3c64cbff28f40ca10a"},"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of ExtraTreesClassifier with train-test split:\")\n\nclf_ExtraTreesClassifier = ExtraTreesClassifier(n_estimators=10, random_state=42)\nclf_ExtraTreesClassifier.fit(xtrain, ytrain)\npredictions = clf_ExtraTreesClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['ExtraTreesClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['ExtraTreesClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['ExtraTreesClassifier']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44e92702e4402c6989e27f2fc0ce5e7e8bd17fea"},"cell_type":"markdown","source":"### RandomForestClassifier"},{"metadata":{"trusted":true,"_uuid":"5a9c4b6f0166d40781845a7aa68c50b77f022269"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of RandomForestClassifier with train-test split:\")\n\nclf_RandomForestClassifier = RandomForestClassifier(n_estimators = 10, random_state=42)\nclf_RandomForestClassifier.fit(xtrain, ytrain)\npredictions = clf_RandomForestClassifier.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['RandomForestClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['RandomForestClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['RandomForestClassifier']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c30dfe3518eec7845dbd522644030299feb321e"},"cell_type":"markdown","source":"### LinearSVC (ovr)"},{"metadata":{"trusted":true,"_uuid":"de643712fcdeff47d49ce6b68765b329847029b6"},"cell_type":"code","source":"from sklearn.svm import LinearSVC\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of LinearSVC, multi_class='ovr', with train-test split:\")\n\nclf_LinearSVC = LinearSVC(multi_class='ovr', random_state=42)\nclf_LinearSVC.fit(xtrain, ytrain)\npredictions = clf_LinearSVC.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['LinearSVC']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['LinearSVC']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['LinearSVC']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a62fe44178f6f74b1824c312e9e4b42e047d28f0"},"cell_type":"markdown","source":"### BernoulliNB"},{"metadata":{"trusted":true,"_uuid":"783dd7bff29b7bb72face33b69f8cf2b0ca091ea"},"cell_type":"code","source":"from sklearn.naive_bayes import BernoulliNB\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of BernoulliNB with train-test split:\")\n\nclf_BernoulliNB = BernoulliNB()\nclf_BernoulliNB.fit(xtrain, ytrain)\npredictions = clf_BernoulliNB.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['BernoulliNB']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['BernoulliNB']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['BernoulliNB']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"307e770149736f860c67d957289ec0a399b83c74"},"cell_type":"markdown","source":"### MultinomialNB - GaussianNB\n I can not use GaussianNB and MultinomialNB because the averaged word embeddings contain and negative numbers and both models can not fit with negative data."},{"metadata":{"_uuid":"c1c691983c031967abd88283e4e0affadf69e71d"},"cell_type":"markdown","source":"### AdaBoostClassifier"},{"metadata":{"trusted":true,"_uuid":"e4d05ed347c8107b55bdedfa0fc6121b839bcecc"},"cell_type":"code","source":"from sklearn.ensemble import AdaBoostClassifier\n\nstart_time = time.time()\n    \nprint()\nprint(\"Evaluation of Adaboost with train-test split:\")\n\nclf_adaboost = AdaBoostClassifier(random_state=42)\nclf_adaboost.fit(xtrain, ytrain)\npredictions = clf_adaboost.predict(xvalid)\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy score\", accuracy_score(yvalid, predictions))\nprint(\"missclass score\", 1 - accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['AdaboostClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['AdaboostClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['AdaboostClassifier']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\n\n### storing performance results:\n\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43eb42a9bc71d1933e83a380b5d387ee4388e7fe"},"cell_type":"markdown","source":"### Xgboost"},{"metadata":{"trusted":true,"_uuid":"960f241ba0d48b581c246c36360ef1a2ee1f2e0d"},"cell_type":"code","source":"import xgboost as xgb\n\nstart_time = time.time()\n\nclf_xgb = xgb.XGBClassifier(objective = 'multi:softmax', seed=42)\nclf_xgb.fit(np.array(xtrain), ytrain)\npredictions = clf_xgb.predict(np.array(xvalid))\n\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['XGB']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['XGB']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['XGB']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"497596f8cb3f8755e8eb8c261c475b66d01b7d09"},"cell_type":"markdown","source":"### KNeighborsClassifier"},{"metadata":{"trusted":true,"_uuid":"b67b4961ffface41f086a7ef3b63241c6cee4d14"},"cell_type":"code","source":"from sklearn.neighbors import KNeighborsClassifier\n\nstart_time = time.time()\n\nclf_knn = KNeighborsClassifier()\nclf_knn.fit(xtrain, ytrain)\npredictions = clf_knn.predict(xvalid)\n\nprint(classification_report(yvalid, predictions))\n\nprint()\nprint(\"accuracy_score\", accuracy_score(yvalid, predictions))\n\nprint()\nprint(\"Weighted Averaged validation metrics\")\nprint(\"precision_score\", precision_score(yvalid, predictions, average='weighted'))\nprint(\"recall_score\", recall_score(yvalid, predictions, average='weighted'))\nprint(\"f1_score\", f1_score(yvalid, predictions, average='weighted'))\n\n\nml_default_performance_metrics_df.loc['KNeighborsClassifier']['training-time'] = time.time() - start_time\nml_default_performance_metrics_df.loc['KNeighborsClassifier']['accuracy'] = accuracy_score(yvalid, predictions)\nml_default_performance_metrics_df.loc['KNeighborsClassifier']['F1-score'] = f1_score(yvalid, predictions, average='macro')\n\nprint()\nfrom sklearn.metrics import confusion_matrix\nimport scikitplot as skplt\nsns.set(rc={'figure.figsize':(8,8)})\nskplt.metrics.plot_confusion_matrix(yvalid, predictions)\n\nprint()\nprint(\"elapsed time: \", time.time() - start_time)\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"35396628e90b834b9359d24de6a6cbb1e7d7dae7"},"cell_type":"markdown","source":"### Summarizing ML Classifiers based on their accuracy with TF - IDF as Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"35a7cd3d7f7b87bc64a7b945f2beb37083b02ea3"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8053a2327f656f28a31d83c1cbfbb37e300c4e7a"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\nml_default_performance_metrics_df.sort_values(by=\"accuracy\", ascending=False).accuracy.plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"413ac13551767e11616a6e73a9d2e8d7425bb181"},"cell_type":"markdown","source":"### Summarizing ML Classifiers based on their F1-score with TF - IDF as Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"f5b34b111b35975755cedf2c8cd0fc467697205f"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"F1-score\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"474c94355e7b55c0831bca2ff470cd52da1ae9ee"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\nml_default_performance_metrics_df.sort_values(by=\"F1-score\", ascending=False)[\"F1-score\"].plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f0f3ab68f15e997c378dc4853cdfbb44faedf1c"},"cell_type":"markdown","source":"### Summarizing ML Classifiers based on their training fitting time with TF - IDF as Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"4be84e26ab9a5e1beb658869e88ec679939febc4"},"cell_type":"code","source":"ml_default_performance_metrics_df.sort_values(by=\"training-time\", ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0173408fee250d80b799dcc9e9b1d061af868c58"},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15.27,6.27)})\nml_default_performance_metrics_df.sort_values(by=\"training-time\", ascending=True)[\"training-time\"].plot(kind=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"061c8b376670d17752d0954c40ff6ff96680104c"},"cell_type":"markdown","source":"## ML Predictions over the Test Set using the Top 3 ML models\nAfter all these experiments there is one conclusion, We cannot combine and use together averages word embeddings and Machine Learning models for this dataset. All ML models performed very bad against themselves and against XGBoost. They missclassify most of cases as class #2 type neutral class and their accuracy is close to 0.5 which means that they fail to classify half of the cases from the validation set and F1-score is even below 0.5 which validates the high missclassification.\n\nThe Top performed ML models in both accuracy, training time and F1-score combined with averaged word embeddings per phrase are the following: \n- ExtraTree\n- ExtraTrees\n- DecisionTrees\n- Random Forest\n\nThere is no need for ensemble models using the statistical mode over the predicted class neither the tuning cause they will not improve dramatically the models' performance.\n\nSo we will move on by producing the predictions over the test set with the top 4 ML models and create submission csv files."},{"metadata":{"trusted":true,"_uuid":"963b1df7f352ae720e83eb228f0bce16588fcff2"},"cell_type":"code","source":"#averaging word embeddings for the test set\n\ndf_test = pd.read_csv(\"../input/test.tsv\", sep=\"\\t\")\nsentences_test = df_test.Phrase.values\n\nmajor_sent_test = []\n\nfor sent in sentences_test:\n    \n    temp_sent = []\n    \n    for word in sent.split(\" \"):    \n        \n        if word in glove_w2v_embeddings_index:\n            temp = glove_w2v_embeddings_index['word']\n        else:\n            temp = np.zeros(embedding_dim)\n\n        temp_sent.append(temp)\n    \n    temp_sent = np.mean(temp_sent, axis=0)\n    major_sent_test.append(temp_sent)\n\ntest_vectorized = major_sent_test\n\n# producing predictions over the test set with the top 4 ML models\n\npredictions_default_extra_trees = clf_ExtraTreesClassifier.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_extra_trees\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_extra_trees.csv',index=False)\n\npredictions_default_extra_tree = clf_ExtraTreeClassifier.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_extra_tree\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_extra_tree.csv',index=False)\n\npredictions_default_decision_tree = clf_DecisionTreeClassifier.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_decision_tree\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_decision_tree.csv',index=False)\n\npredictions_default_random_forest = clf_RandomForestClassifier.predict(test_vectorized)\nsubmission = pd.DataFrame()\nsubmission['PhraseId'] = df_test.PhraseId\nsubmission['Sentiment'] = predictions_default_random_forest\n#submission['Sentiment'] = submission.Sentiment.astype(int)\nsubmission.to_csv('submission_random_forest.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75049d663ae4d96d160dfadae3bc00d8e7a7c03a"},"cell_type":"markdown","source":"## Summary\nThe combination of Word Embeddings and Machine Learning models does not seem to work. **All models** to fit on the data. There is no understanding why this is happening and all models presents accuracy close to 0.5. Even ensembling predictions using the statistical mode was not produced because there no need to due to these unsatisfactory results. It is believed that ML models in general do not pocess the ability to fit on the data. So the time for Deep Learning comes to see if has the power to fit better than Machine Learning.\n___________________"},{"metadata":{"_uuid":"a74b4df01c678a89730431bfe3a24a2b1793f822"},"cell_type":"markdown","source":"___________________________________________________________________"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}