{"cells":[{"metadata":{"_uuid":"e0669b29c1c4d995b15ed075e0cb2b2705a5faca"},"cell_type":"markdown","source":"# NLP Capstone Project - Movie Review - Sentiment Analysis -  Classify the sentiment of sentences from the Rotten Tomatoes dataset\n## Rotten Tomatoes Reviews Analysis - Classify the sentiment of sentences from the Rotten Tomatoes dataset \n\n\n![](https://cdn.steemitimages.com/DQmQZCf7ME7Haj3X3MzXtG8R8JtGmTpuh5NXDSd3wKueva7/rottentomatoes.png)\n\n![Sentiment](https://www.kdnuggets.com/images/sentiment-fig-1-689.jpg)\n\n![EDA](http://www.statistika.co/images/services/Exploratory%20Data%20Analysis%20-%20EDA%201000x468.jpg)"},{"metadata":{"_uuid":"ae81c10ef480ac4f9d24d8519d5602d822fc3afb"},"cell_type":"markdown","source":"# Exploratory Data Analysis\n\nThis Kernel will be the first that will introduce us to the dataset and will try to investigate its features The main strategy that will be followed is the following:\n- Loading necessary libraries\n- Introductory Dataset investigation\n- Observing possible anomalies in the dataset\n- Cleaning the Train Set from Noise\n- Counting Word Frequencies from Unigrams, Bigrams and Trigrams\n- Creating Wordclouds\n- Identifying Named Entities\n- Applying Unsupervized Learning for the purpose of \n    - Train set reviews’ visualization over the 2-axis using t-SNE\n    - K-means clustering over the reviews from train set and visualize the clusters using t-SNE.\n    - Topic Detection over the reviews from train set using LDA (Latent Dirichlet Allocation algorithm) and visualize the topics using t-SNE\n    - Word Embeddings over the train set and visualize their similarity using t-SNE.\n    \n    "},{"metadata":{"_uuid":"e434f86aea4f4b47ccd6b678de73e3e307ef1696"},"cell_type":"markdown","source":"## Load Libraries\nNumerous libraries from data handling to text processing and finally to data vizualization. To be more specific:\n\nLibraries for Data handling:\n- pandas\n- numpy\n\nLibraries for Text processing:\n- sklearn\n- nltk\n- gensim\n\nLibraries for Vizualization:\n- matplotlib\n- seaborn\n- bokeh\n- plotly"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\n\nimport matplotlib\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport seaborn as sns\n\nimport nltk\n\nimport plotly\nimport plotly.graph_objs as go\n\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nimport plotly.graph_objs as go\ninit_notebook_mode(connected=True)\nplotly.offline.init_notebook_mode(connected=True)\n\nfrom wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom bokeh.io import output_file,show,output_notebook,push_notebook\nfrom bokeh.plotting import figure\nfrom bokeh.models import ColumnDataSource,HoverTool,CategoricalColorMapper\nfrom bokeh.layouts import row,column,gridplot\nfrom bokeh.models.widgets import Tabs,Panel\noutput_notebook()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78ea4aa926d80795654089d08a226577af3f38af"},"cell_type":"code","source":"df = pd.read_csv(\"../input/train.tsv\", sep=\"\\t\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6413ae8656b5b890e3b4638477fd021764c01560"},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"523b4477dff595cf310b1fcd6870082362aad78b"},"cell_type":"code","source":"df.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60c95f2353a0f9cb4ddde9c9641b0375a39b2211"},"cell_type":"markdown","source":"PhraseId and SentenceId do not play any role in the analysis since they are both identifiers and incremental numeric features."},{"metadata":{"trusted":true,"_uuid":"a86a51d1f7b9f23a2fb4a006aff43a0673b8753a"},"cell_type":"code","source":"df.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a3c26d3d8fe00b6a493d8f29a033a201cb72e437"},"cell_type":"markdown","source":"Our dataset does not contain missing values."},{"metadata":{"_uuid":"f21dd197db0e4c0751a04a09118b402041266ee2"},"cell_type":"markdown","source":"### Observing the Train Set for potential anomalies:\nHere are couple of instances where punctuations appeared to be predictive."},{"metadata":{"trusted":true,"_uuid":"ace44f8c3558311406b3323e9c2d9f573435c91c"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 0) & (df['PhraseId'] <= 2)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"293b6bf508b75d68368963b793bc924d888a2b0c"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 517) & (df['PhraseId'] <= 518)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"155b327f063abf344671bc3e167bd82dae29623d"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \",\" is important"},{"metadata":{"trusted":true,"_uuid":"8bcf899c1d960ce2f1f7923f3b1d9d7825f7b1f1"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 68) & (df['PhraseId'] <= 69)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"62ff56a99e6fa7305feefb7e5ed1e73a58b4d72a"},"cell_type":"markdown","source":"Below another example that the appearance punctuation symbol \"!\" is important"},{"metadata":{"trusted":true,"_uuid":"30809096f8262921a4110e9e6682c10baa4f1ceb"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 10737) & (df['PhraseId'] <= 10738)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"914201efccdd1f584af4c784c3930472df5224f8"},"cell_type":"markdown","source":"Another strange thing that I discovered is that there are phrases with a single word only and if they disappear at the following phrases the sentiment changes."},{"metadata":{"trusted":true,"_uuid":"1857e6f1f53efd595fa15ee9bbb4093e0b6a32ed"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 22) & (df['PhraseId'] <= 24)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])\n\nprint()\n\nprint(example[\"Phrase\"].values[2], \" - Sentiment:\", example[\"Sentiment\"].values[2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"97394965673dc3e4193a9bb6c733a0769c3290bd"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 46) & (df['PhraseId'] <= 47)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a49a52d7e04c95dafb41b91accfe8cf6472ae869"},"cell_type":"markdown","source":"Another peculiar event that I discovered in the phrases is that they contain words like ** -LRB- and -RRB- **"},{"metadata":{"trusted":true,"_uuid":"21f869eb45a766247c913ec42aea085031433c4b"},"cell_type":"code","source":"example = df[(df['PhraseId'] >= 1225) & (df['PhraseId'] <= 1227)]\n\nprint(example[\"Phrase\"].values[0], \" - Sentiment:\", example[\"Sentiment\"].values[0])\n\nprint()\n\nprint(example[\"Phrase\"].values[1], \" - Sentiment:\", example[\"Sentiment\"].values[1])\n\nprint()\n\nprint(example[\"Phrase\"].values[2], \" - Sentiment:\", example[\"Sentiment\"].values[2])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68a95011fb265fa8117f1e89fb1fa8a6227539b7"},"cell_type":"markdown","source":"### Thoughts about this anomalies\nDuring EDA I will continue the cleaning process in order to unvail some hidden information from phrases of the reviews but I believe that later during the deployment of Machine Learning and Deep Learning models if I cleanup the trainSet I will lose some predictive power / predictiveness for my models. I will come back to this matter later during the Machine Learning and Deep Learning process."},{"metadata":{"_uuid":"2ab2662dff61280d7753460e288f9f6fc512ada8"},"cell_type":"markdown","source":"## Function to create a vocabulary for the phrases from Train Set\nBellow there is a small function which calculates the size of the vocabulary in the Train Set."},{"metadata":{"_uuid":"a763be12be07a9cb5caab180879db44a518dc626"},"cell_type":"markdown","source":"### Tokenizing - Slicing down the sentences into words"},{"metadata":{"trusted":true,"_uuid":"438f9ab427ea5d9e60e778b6c5803a7cbeb1b977"},"cell_type":"code","source":"def tokenize_the_text(phrases):\n    \n    from nltk.tokenize import word_tokenize\n    from nltk.text import Text\n    \n    tokens = [word for word in phrases]\n    tokens = [word.lower() for word in tokens]\n    tokens = [word_tokenize(word) for word in tokens]\n    \n    return tokens\n\ncrude_tokens = tokenize_the_text(df.Phrase)\nprint(crude_tokens[0:10])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0ddab2f0aa4b3214c649c3ca165bef199fb6d3e"},"cell_type":"code","source":"def create_a_vocab(tokens):\n    \n    vocab = set()\n\n    for setence in tokens:\n        for word in setence:\n            vocab.add(word)\n\n    vocab = list(vocab)\n\n    return vocab\n    \nvocab = create_a_vocab(crude_tokens)\n\nprint(\"Vocabulary size:\", len(vocab), \"words\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd5bbc4248ff842a5b8c93bad801f396b92bf25d"},"cell_type":"markdown","source":"## Sentiment Distribution"},{"metadata":{"trusted":true,"_uuid":"2442764a7f33baed49d10443b9b70aa47d676b55"},"cell_type":"code","source":"data = [go.Bar(\n            x=['negative', 'somewhat negative', 'neutral', 'somewhat positive', 'positive'],\n            y=df.Sentiment.value_counts().sort_index(), marker=dict(\n                color = df.Sentiment.value_counts().sort_index(),colorscale='Viridis',showscale=True,\n                reversescale = False\n                ))]\n\nplotly.offline.iplot(data, filename='sentiment distribution')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffb7ae5e2aeaed6785c5dc94ab4ae1809d197d7a"},"cell_type":"code","source":"layout = go.Layout(\n    title='Phrases Length Distribution',\n    xaxis=dict(\n        title='Phrases\\' Length in characters',\n        titlefont=dict(\n            family='Courier New, monospace',\n            size=18,\n            color='#7f7f7f'\n        )\n    ),\n    yaxis=dict(\n        title='Frequencies',\n        titlefont=dict(\n            family='Courier New, monospace',\n            size=18,\n            color='#7f7f7f'\n        )\n    )\n)\n\ndata = [go.Histogram(x=df['Phrase'].apply(lambda x: len(x.split())))]\n\nfig = go.Figure(data=data, layout=layout)\n\nplotly.offline.iplot(fig, filename='sentiment distribution')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69667f7fe828503cbb5c55d72b147f9f2b040a15"},"cell_type":"markdown","source":"## Most Frequent Unigrams Words before cleaning"},{"metadata":{"trusted":true,"_uuid":"c9ad8a094af874c4d671073246299c2463662bb3"},"cell_type":"code","source":"def get_word_count_dict(tokens):\n    \n    words_count_dict = dict()\n\n    for sentence in tokens:\n        for word in sentence:\n            if word not in words_count_dict:\n                words_count_dict[word] = 1\n            else:\n                words_count_dict[word] += 1\n    \n    return words_count_dict\n\n\nwords_count_dict = get_word_count_dict(crude_tokens)\n\nimport operator\n\ntop_uncleaned_words_df = pd.DataFrame()\n\ntop_uncleaned_words_df = top_uncleaned_words_df.append(list(sorted(words_count_dict.items(), key=operator.itemgetter(1), reverse=True)))\ntop_uncleaned_words_df.columns = ['word', 'frequency']\n#print(top_uncleaned_words_df.head(10))\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_uncleaned_words_df.head(limit).word,\n            y=top_uncleaned_words_df.head(limit).frequency, marker=dict(\n                color = top_uncleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                ))\n\nlayout = dict(title= 'Most Frequent words from train set')\n\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2957bf14820a5c5d8a05bff3e9070bd22f8c496b"},"cell_type":"markdown","source":"We can see that very common words such as \"the\", \"a\" \"and\" etc, appears to be the most common words in the Train Set. Having such a big volume of noise in the Phrases from the reviews can hinders us to find better and more tangible words that actually the people uses. This means that text cleaning for Exploratory Data Analysis is a necessity."},{"metadata":{"_uuid":"6b4946d9d399e5b825dac7a1f7f69d5c4d990f90"},"cell_type":"markdown","source":"## Most Frequent Bigrams before text cleaning in the Train Set\nLets investigrate the Bigrams, Bigrams are set of 2 words in the phrases of the reviews. Lets see if by countring their frequency can derive some information."},{"metadata":{"trusted":true,"_uuid":"ecc6fc3684ae5f75a85b6d201ae63e373ec43d4e"},"cell_type":"code","source":"bigram_dict = dict()\nlimit = 20\n\nbigrm = list(nltk.bigrams(' '.join(df.Phrase.values).split()))\n\nfor elem in bigrm:\n    new_elem = ' '.join(elem)\n    if new_elem not in bigram_dict:\n        bigram_dict[new_elem] = 1\n    else:\n        bigram_dict[new_elem] += 1  \n    \ntop_uncleaned_bigram_df = pd.DataFrame.from_dict(bigram_dict, orient='index', columns=[\"frequency\"])\ntop_uncleaned_bigram_df[\"bigram\"] = top_uncleaned_bigram_df.index \ntop_uncleaned_bigram_df = top_uncleaned_bigram_df.reset_index(drop=True)\ntop_uncleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit)\n\n\ntrace1 = go.Bar(\n            y=top_uncleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).frequency,\n            x=top_uncleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).bigram,\n    marker=dict(\n                color = top_uncleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent Bigrams before cleaning')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)\n\ntop_uncleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"376a543685c0f4f7814203e222b580d1c77b698e"},"cell_type":"markdown","source":"We can see that still very common words such as \"the\", \"a\" \"and\" etc, appears to be the most common words in the Train Set. However we can see now that instead of theses noizy words the word \"movie\" and \"film\" start to appear in high frequencies. This means that these words may play a vital use in phrases and this may be discovered after cleaning."},{"metadata":{"_uuid":"bc815dc3f53f3f8016461694172612e159b7cfa3"},"cell_type":"markdown","source":"## Most Frequent Trigrams before text cleaning in the Train Set\nLets investigrate the Trigrams, Bigrams are set of 3 words in the phrases of the reviews. Lets see if by countring their frequency can derive some information."},{"metadata":{"trusted":true,"_uuid":"765c4bb17daac3a91942bfcf2f7f98e60a9cb000"},"cell_type":"code","source":"trigram_dict = dict()\nlimit = 20\n\ntrigrm = list(nltk.trigrams(' '.join(df.Phrase.values).split()))\n\nfor elem in trigrm:\n    new_elem = ' '.join(elem)\n    if new_elem not in trigram_dict:\n        trigram_dict[new_elem] = 1\n    else:\n        trigram_dict[new_elem] += 1\n\n\ntop_uncleaned_trigram_df = pd.DataFrame.from_dict(trigram_dict, orient='index', columns=[\"frequency\"])\ntop_uncleaned_trigram_df[\"bigram\"] = top_uncleaned_trigram_df.index \ntop_uncleaned_trigram_df = top_uncleaned_trigram_df.reset_index(drop=True)\ntop_uncleaned_trigram_df.sort_values(ascending=False, by=\"frequency\").head(limit)\n\n\ntrace1 = go.Bar(\n            y=top_uncleaned_trigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).frequency,\n            x=top_uncleaned_trigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).bigram,\n    marker=dict(\n                color = top_uncleaned_trigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent Trigrams before cleaning')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)\n\ntop_uncleaned_trigram_df.sort_values(ascending=False, by=\"frequency\").head(limit)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f93a80019d9ec79fd607cadd63fd61181cf70a1d"},"cell_type":"markdown","source":"We can see and in trigrams that still very common words such as \"the\", \"a\" \"and\" etc, appears to be the most common words in the Train Set. However we can see now that instead of theses noizy words the word \"movie\" and \"film\" start to appear in high frequencies. This means that these words may play a vital use in phrases and this may be discovered after cleaning. Finally punctuation appears in high frequencies and they must be removed during Text cleaning."},{"metadata":{"_uuid":"7907d935df753cfeb4e126d4e51d0ab5f8fe475e"},"cell_type":"markdown","source":"## Text Cleaning\nThe process that will be followed during cleaning is the following:\n1. Remove redudant space, custom word simplification and removing punctuation\n2. Remove Stopwords\n3. Lemmatize the Phrases"},{"metadata":{"_uuid":"c35c8ae569626aaa61a073f27d4934862047da64"},"cell_type":"markdown","source":"### Define a function to clean the text from noizy text data such as: \n\n- trimming spacing\n- removing redudant punctuation\n- substituting text to a plain form e.g.: won't -> will not"},{"metadata":{"trusted":true,"_uuid":"a8ffdf9d623eddf2d0264156365044a4d7d6f1a7"},"cell_type":"code","source":"def custom_initial_clean(df):\n\n    phrases_X = df.Phrase.copy()\n\n    phrases_X = phrases_X.str.replace('\\\\*', ' ', regex=True)\n    phrases_X = phrases_X.str.replace('\\\\/', ' ', regex=True)\n    phrases_X = phrases_X.str.replace('\\\\\\\\', ' ', regex=True)\n    phrases_X = phrases_X.str.replace('-', ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r'/', ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r'``', ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r'`', ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\"''\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\",\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\"\\.$\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\":\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\"# \", '#', regex=True)\n    phrases_X = phrases_X.str.replace(r\";\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\"?\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\"=\", ' ', regex=True)\n    phrases_X = phrases_X.str.replace(\"...\", ' ', regex=False)\n    phrases_X = phrases_X.str.replace(\"..\", ' ', regex=False)\n\n    phrases_X = phrases_X.str.replace(r'LRB', ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r'RRB', ' ', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[C|c]a n't\", 'cannot', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[W|w]o n't\", 'will not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[W|w]ere n't\", 'were not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[W|w]as n't\", 'was not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[W|w]ould n't\", 'would not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[D|d]oes n't\", 'does not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[I|i]s n't\", 'is not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[C|c]ould n't\", 'could not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[D|d]id n't\", 'did not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[H|h]as n't\", 'has not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[H|h]ave n't\", 'have not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[D|d]o n't\", 'do not', regex=True)\n    phrases_X = phrases_X.str.replace(r\"[A|a]i n't\", \"not\", regex=True)\n    phrases_X = phrases_X.str.replace(r\"[N|n]eed n't\", \"need not\", regex=True)\n    phrases_X = phrases_X.str.replace(r\"[A|a]re n't\", \"are not\", regex=True)\n    phrases_X = phrases_X.str.replace(r\"[S|s]hould n't\", \"should not\", regex=True)\n    phrases_X = phrases_X.str.replace(r\"[H|h]ad n't\", \"had not\", regex=True)\n\n    phrases_X = phrases_X.str.replace(\" 's\", \" \", regex=False)\n    phrases_X = phrases_X.str.replace(\"'s\", \"\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'ve\", \"have\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'d\", \"would\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'ll\", \"will\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'m\", \"am\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'n\", \"and\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'re\", \"are\", regex=False)\n    phrases_X = phrases_X.str.replace(\"'til\", \"until\", regex=False)\n    phrases_X = phrases_X.str.replace(\" ' \", \" \", regex=False)\n    phrases_X = phrases_X.str.replace(\" '\", \" \", regex=False)\n\n    phrases_X = phrases_X.str.replace(r'[ ]{2,}', ' ', regex=True)\n\n    return phrases_X\n\nphrases_X = custom_initial_clean(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd44c3a615660216dee0fb1298c5c29f9d9407cf"},"cell_type":"code","source":"def tokenize_the_text(phrases):\n    \n    from nltk.tokenize import word_tokenize\n    from nltk.text import Text\n    \n    tokens = [word for word in phrases]\n    tokens = [word.lower() for word in tokens]\n    tokens = [word_tokenize(word) for word in tokens]\n    \n    return tokens","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0faf0038cdbeab82e4528714fe09b7c17abc2c0f"},"cell_type":"code","source":"tokens_custom_cleaned = tokenize_the_text(phrases_X)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e56d41afb8761d3c31213171d6c34da2d4e48042"},"cell_type":"markdown","source":"### Calculating new vocabulary after cleaning\nLets see after custom initial Text cleaning in the Train Set how many words are left now."},{"metadata":{"trusted":true,"_uuid":"0f73800106010db595d578df4760b1431090e125"},"cell_type":"code","source":"vocab = create_a_vocab(tokens_custom_cleaned)\n\nprint(\"Vocabulary size after custom cleaning:\", len(vocab), \"words\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93db4ad098df9af3bc98eadf90df68258da79779"},"cell_type":"markdown","source":"### Removing Stopwords\nstop words are words which are filtered out before or after processing of natural language data (text). Though \"stop words\" usually refers to the most common words in a language, there is no single universal list of stop words used by all natural language processing tools, and indeed not all tools even use such a list. Some tools specifically avoid removing these stop words to support phrase search [source](https://en.wikipedia.org/wiki/Stop_words)."},{"metadata":{"trusted":true,"_uuid":"47e63b446aa5e652df33b13b76c93f128808c582"},"cell_type":"code","source":"def removing_stopwords(tokens_custom_cleaned):\n\n    from nltk.corpus import stopwords\n    stop_words = stopwords.words('english')\n    tokens_custom_cleaned_and_without_stopwords = []\n    for sentence in tokens_custom_cleaned:\n        tokens_custom_cleaned_and_without_stopwords.append([word for word in sentence if word not in stop_words])\n        \n    return tokens_custom_cleaned_and_without_stopwords\n\ntokens_custom_cleaned_and_without_stopwords = removing_stopwords(tokens_custom_cleaned)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4151c4364631c1fc378fc4a80e6c62f7dab1fbcc"},"cell_type":"code","source":"vocab = create_a_vocab(tokens_custom_cleaned_and_without_stopwords)\n\nprint(\"Vocabulary size after custom cleaning and removing stopwords:\", len(vocab), \"words\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"abbaab221a130efc72548b65f1be9043fe01c118"},"cell_type":"markdown","source":"### Lemmatizing the Phrases\nThe goal of Lemmatization is to reduce and return forms of words to a common base form. For instance:\n\nam, are, is $\\Rightarrow$ be \ncar, cars, car's, cars' $\\Rightarrow$ car\n\n[source](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html)"},{"metadata":{"trusted":true,"_uuid":"23a535240d34f3f76905cad2967ba03fbd08b775"},"cell_type":"code","source":"def lemmatizing_the_tokens(tokens_custom_cleaned_and_without_stopwords):\n\n    from nltk.stem.wordnet import WordNetLemmatizer \n    lem = WordNetLemmatizer()\n\n    tokens_custom_cleaned_and_without_stopwords_and_lemmatized = []\n\n    for sentence in tokens_custom_cleaned_and_without_stopwords:\n        tokens_custom_cleaned_and_without_stopwords_and_lemmatized.append([lem.lemmatize(word, pos='v') for word in sentence])\n        \n    return tokens_custom_cleaned_and_without_stopwords_and_lemmatized\n\n\ntokens_custom_cleaned_and_without_stopwords_and_lemmatized = lemmatizing_the_tokens(tokens_custom_cleaned_and_without_stopwords)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4c01a1b075d9c64f6b43dd8a2a517df4789e172"},"cell_type":"code","source":"vocab = create_a_vocab(tokens_custom_cleaned_and_without_stopwords_and_lemmatized)\n\nprint(\"Vocabulary size after custom cleaning, removing stopwords and lemmatizing the text:\", len(vocab), \"words\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2acf6418747fa5b3cae802fe487ce31fc4127e1"},"cell_type":"markdown","source":"We can see that amazingly after text cleaning the size of words were reduced from 16540 to 12622!"},{"metadata":{"_uuid":"e4479ded03a62892c174582cc51c34950ed09479"},"cell_type":"markdown","source":"### Longest Words\nLets find out just out of curiocity what are the longest words in the Train Set."},{"metadata":{"trusted":true,"_uuid":"16b5e69ee66d67d45d2d4eb2290f307f44562c8c"},"cell_type":"code","source":"from collections import defaultdict\n\nlongest_words_dict = defaultdict(list)\n\nfor word in vocab:\n        longest_words_dict[str(len(word))].append(word)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a808f8e6264232601f4d4a44d7849bfe9f840b6"},"cell_type":"code","source":"print(\"The Biggest number of characters with the longest words in the Train Set is:\", \n      max([int(elem) for elem in list(longest_words_dict.keys())]))\nprint(longest_words_dict[str(max([int(elem) for elem in list(longest_words_dict.keys())]))])\n\nprint()\n\nprint(\"The Second biggest number of characters with the longest words in the Train Set is:\", \n      max([int(elem) for elem in list(longest_words_dict.keys())]) - 1)\nprint(longest_words_dict[str(max([int(elem) for elem in list(longest_words_dict.keys())])-1)])\n\nprint()\n\nprint(\"The Third biggest number of characters with the longest words in the Train Set is:\", \n      max([int(elem) for elem in list(longest_words_dict.keys())]) - 2)\nprint(longest_words_dict[str(max([int(elem) for elem in list(longest_words_dict.keys())])-2)])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf97a6a1ae1d8985b105a935fd336ecd5bf5cd69"},"cell_type":"markdown","source":"### WorldCloud\nNow that the phrases from the Train Set are cleaned, lets create a worldcloud to vizualize the impact of the frequency of the cleaned words. "},{"metadata":{"trusted":true,"_uuid":"40ad82bd126e4c9c1d0d9558ca20f9b51062906a"},"cell_type":"code","source":"processed_text = [(' '.join(sentence)) for sentence in tokens_custom_cleaned_and_without_stopwords_and_lemmatized]\nwhole_reviews = ' '.join(processed_text)\n\n\n# Create and generate a word cloud image:\nwordcloud = WordCloud(max_font_size=180, max_words=100, width=800, height=600).generate(whole_reviews)\n\n# Display the generated image:\nplt.figure( figsize=(20,10) )\nplt.imshow(wordcloud, interpolation='bilinear')\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67994ef1f1ea6f1ce8751b70f1ed474d68a84632"},"cell_type":"markdown","source":"### Wordcloud per sentiment\nLets do the same but per sentiment"},{"metadata":{"trusted":true,"_uuid":"edbaf2bbb2843bf0a73017405db09ac4425ffcb1"},"cell_type":"code","source":"#np.shape(processed_text)\nprocessed_df = pd.DataFrame(columns=[\"Phrase\", \"Sentiment\"])\nprocessed_df.Phrase = processed_text\nprocessed_df.Sentiment = df.Sentiment.values\n\n\nfig,axes = plt.subplots(3, 2, figsize=(30, 15))\n\ntemp = processed_df[processed_df.Sentiment == 0].Phrase\nwhole_reviews = ' '.join(temp)\n\nwordcloud_1 = WordCloud(max_font_size=180, max_words=100, width=800, height=600).generate(whole_reviews)\n\nax = axes[0, 0]\nax.imshow(wordcloud_1, interpolation=\"bilinear\")\nax.axis('off')\nax.set_title(\"Sentiment 0 - negative, Wordcloud\", fontsize=30)\n\ntemp = processed_df[processed_df.Sentiment == 1].Phrase\nwhole_reviews = ' '.join(temp)\n\nwordcloud_2 = WordCloud(max_font_size=180, max_words=100, width=800, height=600).generate(whole_reviews)\n\nax = axes[0, 1]\nax.imshow(wordcloud_2, interpolation=\"bilinear\")\nax.axis('off')\nax.set_title(\"Sentiment 1 - somewhat negative, Wordcloud\", fontsize=30)\n\ntemp = processed_df[processed_df.Sentiment == 2].Phrase\nwhole_reviews = ' '.join(temp)\n\nwordcloud_3 = WordCloud(max_font_size=180, max_words=100, width=800, height=600).generate(whole_reviews)\n\nax = axes[1, 0]\nax.imshow(wordcloud_3, interpolation=\"bilinear\")\nax.axis('off')\nax.set_title(\"Sentiment 2 - neutral, Wordcloud\", fontsize=30)\n\ntemp = processed_df[processed_df.Sentiment == 3].Phrase\nwhole_reviews = ' '.join(temp)\n\nwordcloud_4 = WordCloud(max_font_size=180, max_words=100, width=800, height=600).generate(whole_reviews)\n\nax = axes[1, 1]\nax.imshow(wordcloud_4, interpolation=\"bilinear\")\nax.axis('off')\nax.set_title(\"Sentiment 3 - somewhat positive, Wordcloud\", fontsize=30)\n\n\ntemp = processed_df[processed_df.Sentiment == 4].Phrase\nwhole_reviews = ' '.join(temp)\n\nwordcloud_5 = WordCloud(max_font_size=180, max_words=100, width=800, height=600).generate(whole_reviews)\n\nax = axes[2, 0]\nax.imshow(wordcloud_5, interpolation=\"bilinear\")\nax.axis('off')\nax.set_title(\"Sentiment 4 - positive, Wordcloud\", fontsize=30)\n\n\naxes[-1, -1].axis('off')\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### Top 20 word Frequencies after text cleaning\nGoing back to word frequencies. Lets view the top 20 word frequencies after text cleaning in the train Set"},{"metadata":{"trusted":true,"_uuid":"57e4840ddac728fb0549e8a15a623b9ea135f17d"},"cell_type":"code","source":"top_cleaned_words_df = pd.DataFrame()\n\ntop_cleaned_words_df = top_cleaned_words_df.append(list(sorted(get_word_count_dict(tokens_custom_cleaned_and_without_stopwords_and_lemmatized).items(), key=operator.itemgetter(1), reverse=True)))\ntop_cleaned_words_df.columns = ['word', 'frequency']\n#print(top_cleaned_words_df.head(10))\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_cleaned_words_df.head(limit).word,\n            y=top_cleaned_words_df.head(limit).frequency,\n    marker=dict(\n                color = top_cleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent words after cleaning')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bee9de9345fb9ebc410c986e74fe2ea450e4c691"},"cell_type":"markdown","source":"After cleaning we can see that more tangible information can be derived. We can see that what words are used from phrases from the movie reviews. "},{"metadata":{"_uuid":"4e463222c54bdd311c86ca43f51d25c4698db81e"},"cell_type":"markdown","source":"After cleaning another question is whats the frequency of words per sentiment after the cleanup of the trainset"},{"metadata":{"_uuid":"77c65d7d2fc807be5a429e2bdf4c9f9e3401ed52"},"cell_type":"markdown","source":"### Top 20 word Frequencies after text cleaning per Sentiment  - **  Sentiment 0 - Negative **\nAfter cleaning another question is whats the frequency of words per sentiment after the cleanup of the trainset"},{"metadata":{"trusted":true,"_uuid":"a77e5d55f308b5da52d3173555a90e579bf846dd"},"cell_type":"code","source":"temp = tokenize_the_text(processed_df[processed_df.Sentiment == 0].Phrase)\ntop_cleaned_words_df = pd.DataFrame()\n\ntop_cleaned_words_df = top_cleaned_words_df.append(list(sorted(get_word_count_dict(temp).items(), key=operator.itemgetter(1), reverse=True)))\ntop_cleaned_words_df.columns = ['word', 'frequency']\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_cleaned_words_df.head(limit).word,\n            y=top_cleaned_words_df.head(limit).frequency,\n    marker=dict(\n                color = top_cleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent words after cleaning for sentiment 0')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9af757d33e6ea886b9daae130e713e40d241862"},"cell_type":"markdown","source":"### Top 20 word Frequencies after text cleaning per Sentiment  - **  Sentiment 1 - somewhat Negative **\nAfter cleaning another question is whats the frequency of words per sentiment after the cleanup of the trainset"},{"metadata":{"trusted":true,"_uuid":"f7abed6bbe32a08ce7e0b1d006a9a2758819c258"},"cell_type":"code","source":"temp = tokenize_the_text(processed_df[processed_df.Sentiment == 1].Phrase)\ntop_cleaned_words_df = pd.DataFrame()\n\ntop_cleaned_words_df = top_cleaned_words_df.append(list(sorted(get_word_count_dict(temp).items(), key=operator.itemgetter(1), reverse=True)))\ntop_cleaned_words_df.columns = ['word', 'frequency']\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_cleaned_words_df.head(limit).word,\n            y=top_cleaned_words_df.head(limit).frequency,\n    marker=dict(\n                color = top_cleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent words after cleaning  for sentiment 1')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2220aebdf4eba1719163a6c0b0650338e77093cd"},"cell_type":"markdown","source":"### Top 20 word Frequencies after text cleaning per Sentiment  - **  Sentiment 2 - Neutral**\nAfter cleaning another question is whats the frequency of words per sentiment after the cleanup of the trainset"},{"metadata":{"trusted":true,"_uuid":"e8d26db3f240646c9da9b80d6746456c827ab3db"},"cell_type":"code","source":"temp = tokenize_the_text(processed_df[processed_df.Sentiment == 2].Phrase)\ntop_cleaned_words_df = pd.DataFrame()\n\ntop_cleaned_words_df = top_cleaned_words_df.append(list(sorted(get_word_count_dict(temp).items(), key=operator.itemgetter(1), reverse=True)))\ntop_cleaned_words_df.columns = ['word', 'frequency']\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_cleaned_words_df.head(limit).word,\n            y=top_cleaned_words_df.head(limit).frequency,\n    marker=dict(\n                color = top_cleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent words after cleaning  for sentiment 2')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ab581090915376b8e9cd03511dbbb60125bedbef"},"cell_type":"markdown","source":"### Top 20 word Frequencies after text cleaning per Sentiment  - **  Sentiment 3 - Somewhat Positive **\nAfter cleaning another question is whats the frequency of words per sentiment after the cleanup of the trainset"},{"metadata":{"trusted":true,"_uuid":"e8ba5e53221c16134d0dee931fc18fa68786d5dd"},"cell_type":"code","source":"temp = tokenize_the_text(processed_df[processed_df.Sentiment == 3].Phrase)\ntop_cleaned_words_df = pd.DataFrame()\n\ntop_cleaned_words_df = top_cleaned_words_df.append(list(sorted(get_word_count_dict(temp).items(), key=operator.itemgetter(1), reverse=True)))\ntop_cleaned_words_df.columns = ['word', 'frequency']\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_cleaned_words_df.head(limit).word,\n            y=top_cleaned_words_df.head(limit).frequency,\n    marker=dict(\n                color = top_cleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent words after cleaning  for sentiment 3')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc5de0e6cf3fd411a4b02cdf94e6134ddf3627c8"},"cell_type":"markdown","source":"### Top 20 word Frequencies after text cleaning per Sentiment  - **  Sentiment 4 - Positive **\nAfter cleaning another question is whats the frequency of words per sentiment after the cleanup of the trainset"},{"metadata":{"trusted":true,"_uuid":"1b155b2de678018004df5da1ae12ec539057d1aa"},"cell_type":"code","source":"temp = tokenize_the_text(processed_df[processed_df.Sentiment == 4].Phrase)\ntop_cleaned_words_df = pd.DataFrame()\n\ntop_cleaned_words_df = top_cleaned_words_df.append(list(sorted(get_word_count_dict(temp).items(), key=operator.itemgetter(1), reverse=True)))\ntop_cleaned_words_df.columns = ['word', 'frequency']\n\nlimit = 20\n\ntrace1 = go.Bar(\n            x=top_cleaned_words_df.head(limit).word,\n            y=top_cleaned_words_df.head(limit).frequency,\n    marker=dict(\n                color = top_cleaned_words_df.head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent words after cleaning for sentiment 4')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8b622683ee081c5d9b2f5499f924be25b3a0e51"},"cell_type":"markdown","source":"### Bigrams after Cleaning\nMovinig on to bigrams, let us find out what set of words are used for review after cleaning."},{"metadata":{"trusted":true,"_uuid":"26e75344e4107010b08c63e9d25ddefa55d70210"},"cell_type":"code","source":"bigram_dict = dict()\nlimit = 20\n\nbigrm = list(nltk.bigrams(whole_reviews.split()))\n\nfor elem in bigrm:\n    new_elem = ' '.join(elem)\n    if new_elem not in bigram_dict:\n        bigram_dict[new_elem] = 1\n    else:\n        bigram_dict[new_elem] += 1\n        \n    \ntop_cleaned_bigram_df = pd.DataFrame.from_dict(bigram_dict, orient='index', columns=[\"frequency\"])\ntop_cleaned_bigram_df[\"bigram\"] = top_cleaned_bigram_df.index \ntop_cleaned_bigram_df = top_cleaned_bigram_df.reset_index(drop=True)\ntop_cleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit)\n\n\ntrace1 = go.Bar(\n            y=top_cleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).frequency,\n            x=top_cleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).bigram,\n    marker=dict(\n                color = top_cleaned_bigram_df.sort_values(ascending=False, by=\"frequency\").head(limit).frequency,colorscale='Cividis',showscale=True,\n                reversescale = False\n                )\n    )\n\nlayout = dict(title= 'Top '+str(limit)+' most frequent Bigrams after cleaning')\nfig=dict(data=[trace1], layout=layout)\nplotly.offline.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb887544a65ff02ab32e333e82fb4c633ae696aa"},"cell_type":"markdown","source":"Text cleaning was very efficient, New set words were emerged such as \"one best\" and \"best film\" that are used frequently for reviewing the movies."},{"metadata":{"_uuid":"0012522e67514d97615d3940a60378e5e7d12d41"},"cell_type":"markdown","source":"### Trigrams after cleaning\nMovinig on to trigrams, let us find out what triplets of words are used for movie reviews after cleaning."},{"metadata":{"trusted":true,"_uuid":"abf7a4f6b68b4f8d2f1729c81fc66540f7e37dfd"},"cell_type":"code","source":"trigram_dict = dict()\n\n\ntrigrm = list(nltk.trigrams(whole_reviews.split()))\n\nfor elem in trigrm:\n    if elem not in bigram_dict:\n        trigram_dict[elem] = 1\n    else:\n        trigram_dict[elem] += 1\n\ntop_cleaned_trigram_df = pd.DataFrame.from_dict(trigram_dict, orient='index', columns=[\"frequency\"])\ntop_cleaned_trigram_df[\"trigram\"] = top_cleaned_trigram_df.index \ntop_cleaned_trigram_df = top_cleaned_trigram_df.reset_index(drop=True)\ntop_cleaned_trigram_df.sort_values(ascending=False, by=\"frequency\").head(10)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6c6fb9b86ad19c8ed954c9d56fa0a575c0066ce"},"cell_type":"markdown","source":"There is nothing to report here, there are no frequent triplets of words that are used in cleaned movie reviews more than once."},{"metadata":{"_uuid":"d58d206e257d1334dcd50e569750c5d12657f76f"},"cell_type":"markdown","source":"### Named Entity Detection / Recognition\nNamed-entity recognition is a task of information extraction that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc [source](https://en.wikipedia.org/wiki/Named-entity_recognition)."},{"metadata":{"trusted":true,"_uuid":"f8279eb8519833c3c2ef22d628615290e043089c"},"cell_type":"code","source":"def get_ne_dict(tokens):\n    \n    from collections import defaultdict\n    ne_dict = defaultdict(list)\n    \n    for sent in tokens_custom_cleaned_and_without_stopwords_and_lemmatized:\n        for entity in nltk.ne_chunk(nltk.pos_tag(sent), binary=False):\n            if len(entity) != 2:\n                ne_dict[entity.label()].append(entity[0][0])\n            \n                \n    return ne_dict\n\nimport operator\n\nne_dict = get_ne_dict(tokens_custom_cleaned_and_without_stopwords_and_lemmatized)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a1a70b585bf97bce18a4f5ad0c2d4fa620a9d601"},"cell_type":"code","source":"for elem in ne_dict.items():\n    print(elem[0],\":\", set(elem[1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"05e6a37256bf8cf64e6166927e5ceb7eaccdd15c"},"cell_type":"markdown","source":"NLTK's Named Entity Recognition function identified the above entities inside the phrases from the Train Set."},{"metadata":{"_uuid":"4a09eee85181af5231d7f051b5e7f6f1460f5234"},"cell_type":"markdown","source":"### Identifying most significant / important words in Phrases from reviews from Train Set using TF-IDF\n\n\ntf-idf is the acronym for Term Frequency–inverse Document Frequency. It quantifies the importance of a particular word in relative to the vocabulary of a collection of documents or corpus. The metric depends on two factors:\n\nTerm Frequency: the occurences of a word in a given document (i.e. bag of words)\nInverse Document Frequency: the reciprocal number of times a word occurs in a corpus of documents\nThink about of it this way: If the word is used extensively in all documents, its existence within a specific document will not be able to provide us much specific information about the document itself. So the second term could be seen as a penalty term that penalizes common words such as \"a\", \"the\", \"and\", etc. tf-idf can therefore, be seen as a weighting scheme for words relevancy in a specific document."},{"metadata":{"trusted":true,"_uuid":"adeb09e50b6e6630180c9aa39a71d5117908a2c2"},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer = TfidfVectorizer(min_df=5, lowercase=True, ngram_range=(1, 2))\n\n\nprocessed_text = [(' '.join(sentence)) for sentence in tokens_custom_cleaned_and_without_stopwords_and_lemmatized]\nvz = vectorizer.fit_transform(processed_text)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ed088935cb18efa82ab11c12f0b0d1dc6873375"},"cell_type":"markdown","source":"vz is a tfidf matrix where:\n\nthe number of rows is the total number of phrases\n\nthe number of columns is the total number of unique tokens across the phrases"},{"metadata":{"_uuid":"296c281ef18392bb15a1244d090ec827a544be58"},"cell_type":"markdown","source":"Below is the 10 tokens with the lowest tfidf score, which is unsurprisingly, very generic words that we could not use to distinguish one description from another."},{"metadata":{"trusted":true,"_uuid":"c950f752bd821b266995f5ea6a4e9ed8caf9c37a"},"cell_type":"code","source":"tfidf = dict(zip(vectorizer.get_feature_names(), vectorizer.idf_))\ntfidf = pd.DataFrame(columns=['tfidf']).from_dict(\n                    dict(tfidf), orient='index')\ntfidf.columns = ['tfidf']\n\ntfidf.sort_values(by=['tfidf'], ascending=True).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5a19db599d0f91bda8712aa1492c09663520951"},"cell_type":"markdown","source":"Below is the 10 tokens with the highest tfidf score, which includes words that are a lot specific that by looking at them"},{"metadata":{"trusted":true,"_uuid":"393101af0c35881e7c508305ca1af81e4fb6d9e9"},"cell_type":"code","source":"tfidf.sort_values(by=['tfidf'], ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a73492bca905feeb608b986781b3707d27fe4606"},"cell_type":"markdown","source":"Lets find out most significant and specialized words per sentiment after cleaning in the phrases of the reviews."},{"metadata":{"_uuid":"6ce0e16ee24f0abb0763e3a15374b68fd3e81d83"},"cell_type":"markdown","source":"### TF - IDF vectorizer per Sentiment  - ** Sentiment 0 - Negative **"},{"metadata":{"trusted":true,"_uuid":"d6d05675598d6149b8efbffc5a13457ca0e4cecc"},"cell_type":"code","source":"temp_tokens = tokenize_the_text(processed_df[processed_df.Sentiment == 0].Phrase)\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer_sent = TfidfVectorizer(min_df=5, lowercase=True, ngram_range=(1, 2))\n\n\nprocessed_text_sent = [(' '.join(sentence)) for sentence in temp_tokens]\nvz_sent = vectorizer_sent.fit_transform(processed_text_sent)\n\ntfidf_sent = dict(zip(vectorizer_sent.get_feature_names(), vectorizer_sent.idf_))\ntfidf_sent = pd.DataFrame(columns=['tfidf']).from_dict(\n                    dict(tfidf_sent), orient='index')\ntfidf_sent.columns = ['tfidf']\n\ntfidf_sent.sort_values(by=['tfidf'], ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc6b0de97a521eefd634fac1e84ffd09e11b85a3"},"cell_type":"markdown","source":"### TF - IDF vectorizer per Sentiment  - ** Sentiment 1 - somewhat negative **"},{"metadata":{"trusted":true,"_uuid":"64adf0c41f1bec6e4c51c595fb141c1b9a3c16e6"},"cell_type":"code","source":"temp_tokens = tokenize_the_text(processed_df[processed_df.Sentiment == 1].Phrase)\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer_sent = TfidfVectorizer(min_df=5, lowercase=True, ngram_range=(1, 2))\n\n\nprocessed_text_sent = [(' '.join(sentence)) for sentence in temp_tokens]\nvz_sent = vectorizer_sent.fit_transform(processed_text_sent)\n\ntfidf_sent = dict(zip(vectorizer_sent.get_feature_names(), vectorizer_sent.idf_))\ntfidf_sent = pd.DataFrame(columns=['tfidf']).from_dict(\n                    dict(tfidf_sent), orient='index')\ntfidf_sent.columns = ['tfidf']\n\ntfidf_sent.sort_values(by=['tfidf'], ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8106a605d819e01d5e048de3b78f1d9a65ba193a"},"cell_type":"markdown","source":"### TF - IDF vectorizer per Sentiment  - ** Sentiment 2 - Neutral ** "},{"metadata":{"trusted":true,"_uuid":"f3911e41df468ed17e65649b3eab30e4f67f86a4"},"cell_type":"code","source":"temp_tokens = tokenize_the_text(processed_df[processed_df.Sentiment == 2].Phrase)\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer_sent = TfidfVectorizer(min_df=5, lowercase=True, ngram_range=(1, 2))\n\n\nprocessed_text_sent = [(' '.join(sentence)) for sentence in temp_tokens]\nvz_sent = vectorizer_sent.fit_transform(processed_text_sent)\n\ntfidf_sent = dict(zip(vectorizer_sent.get_feature_names(), vectorizer_sent.idf_))\ntfidf_sent = pd.DataFrame(columns=['tfidf']).from_dict(\n                    dict(tfidf_sent), orient='index')\ntfidf_sent.columns = ['tfidf']\n\ntfidf_sent.sort_values(by=['tfidf'], ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0eebdd63cd3b10e9ce85012e18c12493001d8602"},"cell_type":"markdown","source":"### TF - IDF vectorizer per Sentiment  - ** Sentiment 3 - Somewhat Positive **"},{"metadata":{"trusted":true,"_uuid":"346176fa1c5c08e74b2eadb381315bc2ae904041"},"cell_type":"code","source":"temp_tokens = tokenize_the_text(processed_df[processed_df.Sentiment == 3].Phrase)\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer_sent = TfidfVectorizer(min_df=5, lowercase=True, ngram_range=(1, 2))\n\n\nprocessed_text_sent = [(' '.join(sentence)) for sentence in temp_tokens]\nvz_sent = vectorizer_sent.fit_transform(processed_text_sent)\n\ntfidf_sent = dict(zip(vectorizer_sent.get_feature_names(), vectorizer_sent.idf_))\ntfidf_sent = pd.DataFrame(columns=['tfidf']).from_dict(\n                    dict(tfidf_sent), orient='index')\ntfidf_sent.columns = ['tfidf']\n\ntfidf_sent.sort_values(by=['tfidf'], ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0de9db275ebfec65f0ff87536deee75bd7392ae7"},"cell_type":"markdown","source":"### TF - IDF vectorizer per Sentiment  - ** Sentiment 4 - Positive **"},{"metadata":{"trusted":true,"_uuid":"920b518b6b9f2994ad9d2c7dd004c180f134832f"},"cell_type":"code","source":"temp_tokens = tokenize_the_text(processed_df[processed_df.Sentiment == 4].Phrase)\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvectorizer_sent = TfidfVectorizer(min_df=5, lowercase=True, ngram_range=(1, 2))\n\n\nprocessed_text_sent = [(' '.join(sentence)) for sentence in temp_tokens]\nvz_sent = vectorizer_sent.fit_transform(processed_text_sent)\n\ntfidf_sent = dict(zip(vectorizer_sent.get_feature_names(), vectorizer_sent.idf_))\ntfidf_sent = pd.DataFrame(columns=['tfidf']).from_dict(\n                    dict(tfidf_sent), orient='index')\ntfidf_sent.columns = ['tfidf']\n\ntfidf_sent.sort_values(by=['tfidf'], ascending=False).head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f16a167a834aaad536b7dd678e1a11aae4e48ca8"},"cell_type":"markdown","source":"_____________________________________\n# Unsupervised Learning\n\n![Unsupervised](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/05/UL.png)"},{"metadata":{"_uuid":"8341f526e84172207130b93a27a1e65746367845"},"cell_type":"markdown","source":"### Vizualizing reviews using TrancatedSVD and t-SNE\nMoving on to Unsupervised Learning, the idea is to vizualize the phrases from reviews of the Train Set in a 2 axis plot. So t-SNE comes in handy here.\n\nt-SNE is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. t-SNE has a cost function that is not convex, i.e. with different initializations we can get different results [source](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html)\n\nI use SVD This transformer performs linear dimensionality reduction by means of truncated singular value decomposition (SVD). Contrary to PCA, this estimator does not center the data before computing the singular value decomposition. This means it can work with scipy.sparse matrices efficiently.\n\nFirst, let's reduce the dimension of the Train Set from TF-IDF vectorizer matrix to n_components (30) using SVD."},{"metadata":{"trusted":true,"_uuid":"b6848932ac1097a83108648141820da9835d4843"},"cell_type":"code","source":"from sklearn.decomposition import TruncatedSVD\n\nn_comp=30\nsvd = TruncatedSVD(n_components=n_comp, random_state=42)\nsvd_tfidf = svd.fit_transform(vz)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"05b706116422e611cf1bd6482f12bee383d3576b"},"cell_type":"markdown","source":"#### t-Distributed Stochastic Neighbor Embedding (t-SNE)\nt-SNE is a technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. The goal is to take a set of points in a high-dimensional space and find a representation of those points in a lower-dimensional space, typically the 2D plane. It is based on probability distributions with random walk on neighborhood graphs to find the structure within the data. But since t-SNE complexity is significantly high, usually we'd use other high-dimension reduction techniques before applying t-SNE.\n\nGiven the high dimension of our tfidf matrix, we need to reduce their dimension using the Singular Value Decomposition (SVD) technique. And to visualize our vocabulary, we could next use t-SNE to reduce the dimension from 30 to 2. t-SNE is more suitable for dimensionality reduction to 2 or 3."},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"8e15fe90fb29ce68e7db6f4b30f1d6414877ef95"},"cell_type":"code","source":"from sklearn.manifold import TSNE\n\ntsne_model = TSNE(n_components=2, verbose=1, random_state=42, n_iter=500)\ntsne_tfidf = tsne_model.fit_transform(svd_tfidf)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10e09a73011c2a1b7e7e4baf0acaf7a28969cb24"},"cell_type":"markdown","source":"It's now possible to visualize our data points. Note that the deviation as well as the size of the clusters imply little information in t-SNE."},{"metadata":{"_uuid":"90458f60e1a683fbd8d8be6e6cc26a579c885560"},"cell_type":"markdown","source":"### Train set reviews’ visualization over the 2-axis using t-SNE."},{"metadata":{"trusted":true,"_uuid":"6b559f0ea7eb56804a84144e12f13f7b79bcfc75"},"cell_type":"code","source":"plot_df = pd.DataFrame(columns=[\"x\", \"y\", \"text\"])\nplot_df.x = tsne_tfidf[:,0]\nplot_df.y = tsne_tfidf[:,1]\nplot_df.text = processed_text\n\nsource = ColumnDataSource(data=dict(x=plot_df['x'], y=plot_df['y'],\n                                    text=plot_df['text']))\n\n\nplot_tfidf = figure(plot_width=700, plot_height=600,\n                        title=\"Tf-IDF text to features representation using tSNE\",\n    tools=\"pan,wheel_zoom,box_zoom,reset,hover,previewsave\",\n    x_axis_type=None, y_axis_type=None, min_border=1)\n\nplot_tfidf.scatter(source = source, x='x', y='y', color='#4286f4')\nhover = plot_tfidf.select(dict(type=HoverTool))\nhover.tooltips={\"text\": \"@text\" }\noutput_file(\"Train_set_reviews_visualization_over_the_2-axis_using_t-SNE.html\")\nshow(plot_tfidf)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd48afb5eb849f7d0c8d2e8360ef2307e402972e"},"cell_type":"markdown","source":"Based on the vizualization above, we can see neighborhoods of phrases, since t-SNE depicts the phrases based on "},{"metadata":{"_uuid":"d0353cf377af332be4a68b4eac86862826e558f9"},"cell_type":"markdown","source":"## Unsupervised Learning - Clustering the reviews with Kmeans and t-SNE vizualization\nK-means clustering obejctive is to minimize the average squared Euclidean distance of the document / description from their cluster centroids. At first lets find the optimal number of clusters that fit to the phrases in the train set and evaluate cluster's with silhouette score. The silhouette score is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The silhouette ranges from −1 to +1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. If most objects have a high value, then the clustering configuration is appropriate. If many points have a low or negative value, then the clustering configuration may have too many or too few clusters [source](https://en.wikipedia.org/wiki/Silhouette_(clustering).\n\nMini batch Kmeans was prefer to Kmeans because the former uses small random batches of examples of a fixed size so they can be stored in memory. Each iteration a new random sample from the dataset is obtained and used to update the clusters and this is repeated until convergence [source](https://www.google.com/search?q=mini+batch+k+means+vs+k+means&oq=mini+batch+k+means+vs+k+means&aqs=chrome..69i57.5712j0j1&sourceid=chrome&ie=UTF-8)."},{"metadata":{"_uuid":"42982fdcc74ade74fb9912908312a38739f7d833"},"cell_type":"markdown","source":"### Kmeans grid search\nFind the optimal number of clusters and evaluate clustering via silhouette score."},{"metadata":{"trusted":true,"_uuid":"ccda2e47b5773a354be2a3964f905216190cda82"},"cell_type":"code","source":"from sklearn.metrics import silhouette_samples, silhouette_score\nfrom sklearn.cluster import MiniBatchKMeans\n\n\nrange_n_clusters = [6, 8, 10, 12, 14, 16, 18]\n\ncluster_grid_df = pd.DataFrame(columns=[\"cluster_size\", \"silhouette_score\"])\n\nfor num_clusters in range_n_clusters:\n    \n    kmeans_model = MiniBatchKMeans(n_clusters=num_clusters,\n                               init='k-means++',\n                               n_init=1,\n                               init_size=1000, batch_size=1000, verbose=0, max_iter=1000, random_state=42)\n\n    kmeans = kmeans_model.fit(vz)\n    kmeans_clusters = kmeans.predict(vz)\n    silhouette_avg = silhouette_score(vz, kmeans_clusters)\n    print(\"For n_clusters =\", num_clusters, \"The average silhouette_score is :\", silhouette_avg)\n    cluster_grid_df = cluster_grid_df.append({'cluster_size': num_clusters, 'silhouette_score': silhouette_avg}, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"abf9d63928e6bace02bd0617e37d6424814fe374"},"cell_type":"code","source":"num_clusters = int(cluster_grid_df[np.abs(cluster_grid_df.silhouette_score) == np.max(cluster_grid_df.silhouette_score)].cluster_size)\n\n\nkmeans_model = MiniBatchKMeans(n_clusters=num_clusters,\n                               init='k-means++',\n                               n_init=1,\n                               init_size=1000, batch_size=1000, verbose=0, max_iter=1000, random_state=42)\n\nkmeans = kmeans_model.fit(vz)\nkmeans_clusters = kmeans.predict(vz)\nkmeans_distances = kmeans.transform(vz)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92f3e1e07155e14a10dfed00751deb4f1fe14786"},"cell_type":"markdown","source":"### Representative terms per cluster center (cluster center)"},{"metadata":{"trusted":true,"_uuid":"1c0e7d5e0dd18a2b86aff80abad1dbacbf062dc9"},"cell_type":"code","source":"num_clusters = int(cluster_grid_df[np.abs(cluster_grid_df.silhouette_score) == np.max(cluster_grid_df.silhouette_score)].cluster_size)\n\nprint(\"Representative terms per cluster center:\")\norder_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]\nterms = vectorizer.get_feature_names()\nfor i in range(num_clusters):\n    print(\"Cluster %d:\" % i)\n    center_cluster_str_words = []\n    for ind in order_centroids[i, :10]:\n        str = \"\".join(terms[ind])\n        center_cluster_str_words.append(str)\n    print(' %s' % \"|\".join(center_cluster_str_words))\n    print()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d71024e74a3adef8b823a358c92f4b508d90eccb","_kg_hide-output":true},"cell_type":"code","source":"from sklearn.manifold import TSNE\ntsne_model = TSNE(n_components=2, verbose=1, init='pca', random_state=42, n_iter=500)\n\ntsne_kmeans = tsne_model.fit_transform(kmeans_distances)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a2558bf0ec1add0c82b18fcd3f8f2ba58c66ef7f"},"cell_type":"code","source":"\ncolormap = np.array([\n    \"#000080\", \"#00BFFF\", \"#008000\", \"#808000\", \"#FF8C00\",\n    \"#FF7F50\", \"#4ba04e\", \"#FF6347\", \"#A9A9A9\", \"#808080\",\n    \"#ADD8E6\", \"#FF00FF\", \"#90EE90\", \"#C0C0C0\", \"#FF00FF\",\n    \"#008080\", \"#4169E1\", \"#BDB76B\", \"#F0FFF0\", \"#F4A460\",\n    \"#4B0082\", \"#FA8072\", \"#9ACD32\", \"#7CFC00\", \"#DDA0DD\",\n    \"#A52A2A\", \"#F5F5DC\", \"#FFEFD5\", \"#008080\", \"#000000\"\n])\n\n\nplot_df = pd.DataFrame(columns=[\"x\", \"y\", \"text\", \"cluster\"])\nplot_df.x = tsne_kmeans[:,0]\nplot_df.y = tsne_kmeans[:,1]\nplot_df.text = processed_text\nplot_df.cluster = kmeans_clusters\n\nsource = ColumnDataSource(data=dict(x=plot_df['x'], y=plot_df['y'],\n                                    cluster=plot_df['cluster'],\n                                    color=colormap[kmeans_clusters],\n                                    text=plot_df['text']))\n\n\nplot_kmeans = figure(plot_width=700, plot_height=600,\n                        title=\"KMeans clustering of the description\",\n    tools=\"pan,wheel_zoom,box_zoom,reset,hover,previewsave\",\n    x_axis_type=None, y_axis_type=None, min_border=1)\n\nplot_kmeans.scatter(source = source, x='x', y='y', color='color')\nhover = plot_kmeans.select(dict(type=HoverTool))\nhover.tooltips={\"text\": \"@text\", \"cluster\":\"@cluster\" }\noutput_file(\"K-means_reviews_clustering_t-SNE_vizualization.html\")\nshow(plot_kmeans)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9dfe8dddd5314e2eb7f42e5a3b2804600def6a4"},"cell_type":"markdown","source":"## Unsupervised Learning - Topic Modeling with the reviews using LDA and t-SNE vizualization\nLatent Dirichlet Allocation (LDA) is an algorithms used to discover the topics that are present in a corpus.\n\nLDA starts from a fixed number of topics. Each topic is represented as a distribution over words, and each document is then represented as a distribution over topics. Although the tokens themselves are meaningless, the probability distributions over words provided by the topics provide a sense of the different ideas contained in the documents.\n\nReference: https://medium.com/intuitionmachine/the-two-paths-from-natural-language-processing-to-artificial-intelligence-d5384ddbfc18\n\nBoth K-means and Latent Dirichlet Allocation (LDA) are unsupervised learning algorithms, where the user needs to decide a priori the parameter K, respectively the number of clusters and the number of topics.\n\nIf both are applied to assign K topics to a set of N documents, the most evident difference is that K-means is going to partition the N documents in K disjoint clusters (i.e. topics in this case). On the other hand, LDA assigns a document to a mixture of topics. Therefore each document is characterized by one or more topics (e.g. Document D belongs for 60% to Topic A, 30% to topic B and 10% to topic E). Hence, LDA can give more realistic results than k-means for topic assignment [source](https://www.quora.com/What-are-the-differences-and-similarities-between-LDA-and-k-means-for-topic-detection-assuming-that-I-can-cluster-documents-with-k-means-and-extract-some-common-key-phrases-to-represent-their-topics).\n\nIts input is a bag of words, i.e. each document represented as a row, with each columns containing the count of words in the corpus."},{"metadata":{"trusted":true,"_uuid":"c80e00093605d9b457d5f1e467ecb61daee372d9"},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\n\ncvectorizer = CountVectorizer(min_df=5, lowercase=True, ngram_range=(1,2))\n\nprocessed_text = [(' '.join(sentence)) for sentence in tokens_custom_cleaned_and_without_stopwords_and_lemmatized]\ncvz = cvectorizer.fit_transform(processed_text)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef8b838811ebbc6902d2d1355b6e8c862ac9b874"},"cell_type":"markdown","source":"### Chossing the right number of topics in LDA\nThe most common way to evaluate a probabilistic model is to measure the log-likelihood or the perplexity of an LDA model. The lower the perplexity, the better the model [source](http://qpleple.com/perplexity-to-evaluate-topic-models/). "},{"metadata":{"trusted":true,"_uuid":"cad09c3c54828c17e146531192552c09a79f675e"},"cell_type":"code","source":"from sklearn.decomposition import LatentDirichletAllocation\n\n# Define Search Param\nlist_of_components= [6,8,10,12]\n\nlda_grid_df = pd.DataFrame(columns=[\"components\", \"perplexity\", \"log_likelihood\"])\n\nfor components in list_of_components:\n    \n    lda_model = LatentDirichletAllocation(n_components=components, max_iter=20, learning_method='online', random_state=42)\n    X_topics = lda_model.fit_transform(cvz)\n    print(\"number of components: \", components)\n    print(\"perplexity: \", lda_model.perplexity(cvz))\n    print(\"log likelihood score: \", lda_model.score(cvz))\n    print()\n    \n    lda_grid_df = lda_grid_df.append({'components': components, 'perplexity': lda_model.perplexity(cvz), \"log_likelihood\": lda_model.score(cvz)}, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82992eef44b9d9b2344344d79afb249585954d7c"},"cell_type":"code","source":"from sklearn.decomposition import LatentDirichletAllocation\n\nn_topics = 12\nn_iter = 20 # number of iterations\n\nlda_model = LatentDirichletAllocation(n_components=n_topics,\n                                      learning_method='online',\n                                      max_iter=n_iter,\n                                      random_state=42)\n\nX_topics = lda_model.fit_transform(cvz)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4c854dac24f7fda7c1faf9571689d5dd9b49a64b"},"cell_type":"markdown","source":"Lets view top keywords per topic"},{"metadata":{"trusted":true,"_uuid":"0f9d49f645eb24bb4cfccd8ab26e8e21e78f6e9a"},"cell_type":"code","source":"n_top_words = 10\ntopic_summaries = []\n\ntopic_word = lda_model.components_  # get the topic words\nvocab = cvectorizer.get_feature_names()\n\nfor i, topic_dist in enumerate(topic_word):\n    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]\n    topic_summaries.append(' '.join(topic_words))\n    print('Topic {}: {}'.format(i, ' | '.join(topic_words)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0d8f10cb1368cb72a4c8f1195413ddc8826235cd","_kg_hide-output":true},"cell_type":"code","source":"# reduce dimension to 2 using tsne\nfrom sklearn.manifold import TSNE\ntsne_model = TSNE(n_components=2, verbose=1, init='pca', random_state=42, n_iter=500)\ntsne_lda = tsne_model.fit_transform(X_topics)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a4baa52c25b827ca3fe0346a515ccbe491108cc1"},"cell_type":"markdown","source":"Now it is time to visualize the topics in the 2-axis using t-SNE"},{"metadata":{"trusted":true,"_uuid":"789d6b3b3b729bd5349fc9c6f9bd55ef4c4191dc","scrolled":false},"cell_type":"code","source":"plot_df = pd.DataFrame(columns=[\"x\", \"y\", \"text\", \"topic\"])\nplot_df.x = tsne_lda[:,0]\nplot_df.y = tsne_lda[:,1]\nplot_df.text = processed_text\nplot_df.topic = X_topics.argmax(axis=1)\n\ncolormap = np.array([\n    \"#000080\", \"#00BFFF\", \"#008000\", \"#808000\", \"#FF8C00\",\n    \"#FF7F50\", \"#a368a3\", \"#FF6347\", \"#A9A9A9\", \"#808080\",\n    \"#ADD8E6\", \"#FF00FF\", \"#90EE90\", \"#C0C0C0\", \"#FF00FF\",\n    \"#008080\", \"#4169E1\", \"#BDB76B\", \"#F0FFF0\", \"#F4A460\",\n    \"#4B0082\", \"#FA8072\", \"#9ACD32\", \"#7CFC00\", \"#DDA0DD\",\n    \"#A52A2A\", \"#F5F5DC\", \"#FFEFD5\", \"#008080\", \"#000000\"\n])\n\nsource = ColumnDataSource(data=dict(x=plot_df['x'], y=plot_df['y'],\n                                    topic=plot_df['topic'],\n                                    color=colormap[plot_df['topic']],\n                                    text=plot_df['text']))\n\n\nplot_lda = figure(plot_width=700, plot_height=600,\n                        title=\"LDA Topic Visualization\",\n    tools=\"pan,wheel_zoom,box_zoom,reset,hover,previewsave\",\n    x_axis_type=None, y_axis_type=None, min_border=1)\n\nplot_lda.scatter(source = source, x='x', y='y', color='color')\nhover = plot_lda.select(dict(type=HoverTool))\nhover.tooltips={\"text\": \"@text\", \"topic\":\"@topic\" }\noutput_file(\"LDA_and_tSNE_visualization.html\")\nshow(plot_lda)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c48b80b0ffe6d280c36934820cfa7e9b205c7ce3"},"cell_type":"markdown","source":"## Word Embeddings over the words in the Train Set using word2vec and tSNE\nLets move on from examining the phrases from the reviews back to words and try to vizualize the words now using again the t-SNE algorithm. Here in order to convert the words into a tangible form the Word Embeddings technique will be used.\n\nWord embedding is one of the most popular feature representation of document vocabulary. It is capable of capturing context of a word in a document, semantic and syntactic similarity, relation with other words\n\nWord embeddings are vector representations of a particular word. Word2Vec is one of the most popular technique to learn word embeddings using shallow neural network. It was developed by Tomas Mikolov in 2013 at Google [source](https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa)."},{"metadata":{"trusted":true,"_uuid":"3507e7fda9f92bca3728166806b05b241daf443d"},"cell_type":"code","source":"import gensim\n\nsentences = [' '.join(sent) for sent in tokens_custom_cleaned_and_without_stopwords_and_lemmatized]\n\n# Creating the model and setting values for the various parameters\nnum_features = 300  # Word vector dimensionality\nmin_word_count = 10 # Minimum word count\nnum_workers = 4     # Number of parallel threads\ncontext = 5        # Context window size\n\n# Initializing the train model\nfrom gensim.models import word2vec\n\n\nprint(\"Training model....\")\nmodel = word2vec.Word2Vec(tokens_custom_cleaned_and_without_stopwords_and_lemmatized,\\\n                          workers=num_workers,\\\n                          size=num_features,\\\n                          min_count=min_word_count,\\\n                          window=context)\n\n# To make the model memory efficient\nmodel.init_sims(replace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"707645a40b2de8e81d944e798bf35b922183924b"},"cell_type":"markdown","source":"### Examples of Similar Words\nLets view the top 4 similar words in some very common words in the phrases from the trainset"},{"metadata":{"trusted":true,"_uuid":"b2591ad5fb406fd5894171338602ecd2f09320b4"},"cell_type":"code","source":"import warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\nprint(\"Top 4 similarities with positive cosine similarity\")\nprint()\n\nprint(\"top 4 similar words for the word film:\", model.wv.most_similar(positive=['film'], topn = 4))\nprint()\n\nprint(\"top 4 similar words for the word film:\",model.wv.most_similar(positive=['great'], topn = 4))\nprint()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8515c1181ff805525a22c5e0d3ba5baf8a093b79"},"cell_type":"markdown","source":"### Convert from Word2vec and t-SNE to 2 and 3 dimensions and illustate the results in 2D and 3D respectively"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"149556370c3d2af6f531c88202678d84e4fbb16d"},"cell_type":"code","source":"from sklearn.manifold import TSNE\n\nlabels = []\nw2v_vectors = []\n\nfor word in model.wv.vocab:\n    w2v_vectors.append(model[word])\n    labels.append(word)\n\ntsne_model = TSNE(n_components=2, init='pca', n_iter=500, random_state=42)\ntsne_w2v = tsne_model.fit_transform(w2v_vectors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"961bcb9478080997222e25b956e582254002529c","scrolled":true},"cell_type":"code","source":"plotly.offline.init_notebook_mode(connected=True)\n\ndata = [go.Scatter(x=tsne_w2v[:,0], y=tsne_w2v[:,1], \n                   mode='markers', text = labels)]\n\nfig = go.Figure(data=data, layout= go.Layout(title=\"Corpus Representation using Word2Vec and tSNE\"))\n\niplot(fig, filename='2D_trainset_vocabulary_representation_using_w2v_and_tSNE')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"d18f2bbf79efa7547d7197f531eb49178e6ecc56"},"cell_type":"code","source":"tsne_model = TSNE(n_components=3, init='pca', n_iter=500, random_state=42)\ntsne_w2v_3d = tsne_model.fit_transform(w2v_vectors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a6783bbe932bd8e36ddb3cecf40d2a8d798e5333"},"cell_type":"code","source":"\ndata = [go.Scatter3d(x=tsne_w2v_3d[:,0], y=tsne_w2v_3d[:,1], z=tsne_w2v_3d[:,2], \n                   mode='markers', text = labels, \n                     marker = dict(size = 4, color = 'rgba(0, 10, 157, .6)'))]\n\nfig = go.Figure(data=data, layout= go.Layout(title=\"3D Corpus Representation using Word2Vec and tSNE\", autosize=False,\n    width=900,\n    height=900))\n\niplot(fig, filename='Corpus_representation_using_w2v_and_tSNE')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"174ceeea7fd0c6374fd349c751606982d77d942d"},"cell_type":"markdown","source":"## EDA and Unsupervized Learning Summary\n- During EDA it was clear and inevitable that text cleaning must be applied in order to discover how the people express from sentiment to sentiment.\n- Furthermore, during EDA various insights have been discovered from word frequencies, bigrams, trigrams, named entities, wordclouds, most relevant words per sentiment.\n- During Unsupervised Learning 2 major dimensionality reduction techniques were applied; SVD and PCA. SVD as a formula to reduce the dimensions from the TF - IDF matrix to 30 dimensions and PCA as a paremeter inside t-SNE algorithm. Phrases / reviews and words visualisations in 2D and 3D were created witht the aid of t-SNE depicting the The phrases in the cartesian system, Kmeans clustering, LDA Topic Modeling and Word Embeddings. \n\nLet us move on the Machine Leaning and Deep Learning model development."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}