{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ✔️ Apply Topic Modeling algorithms using Gensim — before moving on to more advanced textual analysis techniques\n\n> Topic Modelling is a great way to analyze completely unstructured textual data. Python NLP framework such as Gensim, NLTK, and spaCy makes it easier to do this.\n\n> The purpose of this article is to guide one through the whole process of topic modeling right from pre-processing your raw textual data, creating your topic models, evaluating the topic models, to visualizing them. The python packages used during the tutorial will be spaCy (for pre-processing), Gensim (for topic modeling), and pyLDAvis (for visualization).\n\n# 📌 Notebook Goals\n> - Learn how to use the power of `spaCy` to clean textual data.\n> - Use different Topic Modelling techniques like `LDA (Latent Dirichlet Allocation)`, `LSI (Latent Semantic Indexing)`, and `HDP (Hierarchical Drichlet Process)`\n---\n\n# 📚 Topic Modelling Overview\nLet's understand the general concept of Topic Modelling and why it's important! \n> - Topic Modeling is an unsupervised machine learning technique that allows us to efficiently analyze large volumes of text by clustering documents into topics.\n> - A large amount of text data is unlabeled meaning we can’t apply Supervised Learning approaches to create machine learning models for the data! In this case of text data, this means attempting to discover clusters of documents, grouped by topic. A very important idea to keep in mind here is that we don’t know the correct topic or the right answer! All we know is that the documents clustered together share similar topic ideas. It is up to us to identify what these topics represent.\n\n---\n# 📑 Text Analysis Tutorial\n\n> Our steps, naturally, are setting up our imports. We will be using spaCy for data pre-processing and computational linguistics, Gensim for Topic Modeling, Scikit-Learn for classification, and Keras for text generation.","metadata":{}},{"cell_type":"markdown","source":"# ✔️ Import Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\n\nimport spacy \nfrom spacy import displacy\n\nimport gensim\nfrom gensim.corpora import Dictionary\nfrom gensim.models import LdaModel, CoherenceModel, LsiModel, HdpModel\nfrom gensim.models.wrappers import LdaMallet\n\nimport matplotlib.pyplot as plt\nimport sklearn\nimport keras\n\nimport warnings\n\nwarnings.filterwarnings('ignore', category=DeprecationWarning)","metadata":{"execution":{"iopub.status.busy":"2023-02-16T22:13:16.621379Z","iopub.execute_input":"2023-02-16T22:13:16.621844Z","iopub.status.idle":"2023-02-16T22:13:23.866573Z","shell.execute_reply.started":"2023-02-16T22:13:16.621747Z","shell.execute_reply":"2023-02-16T22:13:23.865659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(spacy.__version__)\nprint(gensim.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-02-16T22:13:23.868459Z","iopub.execute_input":"2023-02-16T22:13:23.869055Z","iopub.status.idle":"2023-02-16T22:13:23.875877Z","shell.execute_reply.started":"2023-02-16T22:13:23.869008Z","shell.execute_reply":"2023-02-16T22:13:23.874701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# 📂 Gathering Data\n\n> The dataset we will be working with will be the Lee corpus which is a shortened of the Lee Background Corpus, and the 20NG dataset. ","metadata":{}},{"cell_type":"code","source":"test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])\nprint(test_data_dir)\n\nlee_train_file = test_data_dir + os.sep + 'lee_background.cor'\nprint(lee_train_file)\n\ntext = open(lee_train_file).read()","metadata":{"execution":{"iopub.status.busy":"2023-02-16T22:13:23.877462Z","iopub.execute_input":"2023-02-16T22:13:23.878118Z","iopub.status.idle":"2023-02-16T22:13:23.889907Z","shell.execute_reply.started":"2023-02-16T22:13:23.878073Z","shell.execute_reply":"2023-02-16T22:13:23.889064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# 🧹 Cleaning Data\n\n> We can't have state-of-the-art results without data which is as good. Let's spend this section working on cleaning and understanding our data set. We will be checking out `spaCy`, an industry grade text-processing package.","metadata":{}},{"cell_type":"code","source":"nlp = spacy.load('en')","metadata":{"execution":{"iopub.status.busy":"2023-02-16T22:13:23.891318Z","iopub.execute_input":"2023-02-16T22:13:23.891591Z","iopub.status.idle":"2023-02-16T22:13:24.868149Z","shell.execute_reply.started":"2023-02-16T22:13:23.891563Z","shell.execute_reply":"2023-02-16T22:13:24.867051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> For safe measure, let's add some stopwords. It's a newspaper corpus, so it is likely we will be coming across variations of 'said', 'Mister', and 'Mr'... which will not really add any value to the topic models.","metadata":{}},{"cell_type":"code","source":"my_stop_words = ['say', '\\s', 'mr', 'Mr', 'said', 'says', 'saying', 'today', 'be']\n\nfor stopword in my_stop_words:\n    lexeme = nlp.vocab[stopword]\n    lexeme.is_stop = True","metadata":{"execution":{"iopub.status.busy":"2023-02-16T22:13:24.872712Z","iopub.execute_input":"2023-02-16T22:13:24.873219Z","iopub.status.idle":"2023-02-16T22:13:24.884315Z","shell.execute_reply.started":"2023-02-16T22:13:24.873169Z","shell.execute_reply":"2023-02-16T22:13:24.883044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"doc = nlp(text)","metadata":{"execution":{"iopub.status.busy":"2023-02-16T22:13:24.886391Z","iopub.execute_input":"2023-02-16T22:13:24.886920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# doc","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💹 Computational Linguistics\n\nNow that we have our doc object. We can see that the doc object now contains the entire corpus. This is important because we will be using the doc object to create our corpus for the machine learning algorithms. When creating a corpus for `gensim/scikit-learn`, we sometimes forget the incredible power which `spaCy` packs in its pipeline, so we will briefly demonstrate the same in this section with a smaller example sentence.\n","metadata":{}},{"cell_type":"code","source":"sent = nlp('Last Thursday, Manchester United defeated AC Milan at San Siro.')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🔖 POS-Tagging\n\nThe **Part Of Speech (POS)** explains how a word is used in a sentence. There are eight main parts of speech — nouns, pronouns, adjectives, verbs, adverbs, prepositions, conjunctions, and interjections.","metadata":{}},{"cell_type":"code","source":"for token in sent:\n    print(token.text, token.pos_, token.tag_)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🔖 NER-Tagging  — (Named Entity Recognition)\n\n**Named-entity recognition (NER)** is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.","metadata":{}},{"cell_type":"code","source":"for token in sent:\n    print(token.text, token.ent_type_)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for ent in sent.ents:\n    print(ent.text, ent.label_)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"displacy.render(sent, style='ent', jupyter=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧮 Dependency Parsing\n\nThe term Dependency Parsing (DP) refers to the process of examining the dependencies between the phrases of a sentence in order to determine its grammatical structure.","metadata":{}},{"cell_type":"code","source":"for chunk in sent.noun_chunks:\n    print(chunk.text, chunk.root.text, chunk.root.dep_, chunk.root.head.text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for token in sent:\n    print(token.text, token.dep_, token.head.text, token.head.pos_,\n         [child for child in token.children])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"displacy.render(sent, style='dep', jupyter=True, options={'distance':90})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🧹 Continuing Cleaning\n\n> Have a quick look at the output of the doc object. It seems like nothing, right? But spaCy's internal data structure has done all the work for us. Let's see how we can create our corpus.","metadata":{}},{"cell_type":"code","source":"# We add some words to the stop word list\ntexts, article = [], []\n\nfor word in doc:\n    \n    if word.text != '\\n' and not word.is_stop and not word.is_punct and not word.like_num and word.text != 'I':\n        article.append(word.lemma_)\n        \n    if word.text == '\\n':\n        texts.append(article)\n        article = []\n        \n        \nprint(texts[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> - And this is the magic of spaCy - just like that, we've managed to get rid of stopwords, puctuation markers, and added lemmatized word.\n> - Sometimes topic modeling make more sense when `New` and `York` are treated as `New York` - we can do this by creating a bigram model and modifying our corpus accordingly.","metadata":{}},{"cell_type":"code","source":"bigram = gensim.models.phrases.Phrases(texts)\ntexts = [bigram[line] for line in texts]\ntexts = [bigram[line] for line in texts]\n\nprint(texts[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dictionary = Dictionary(texts)\ncorpus = [dictionary.doc2bow(text) for text in texts]\n\nprint(corpus[1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Now we are done with a very important part of text analysis - the data cleaning and setting up of corpus. It must kept in mind that we created the corpus the way we did because that's how gensim requires it - most algorithms still require one to clean the data set the way we did, by removing stop words and numbers, adding the lemmatized form of the word, and using bigrams.","metadata":{}},{"cell_type":"markdown","source":"---\n# 📚 Topic Modeling\n\n> Topic Modeling refers to the probabilistic modeling of text document as topics. Gensim remains the most popular library to perform such modelling, and we will be using it to perform our topic modelling.","metadata":{}},{"cell_type":"markdown","source":"## ✔️ LSI - Latent Semantic Indexing\n\n> LSI stands for Latent Semantic Indexing - It is a popular information retreival method which works by decomposing the original matrix of words to maintain key topics. ","metadata":{}},{"cell_type":"code","source":"lsi_model = LsiModel(corpus=corpus, num_topics=10, id2word=dictionary)\nlsi_model.show_topics(num_topics=5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ HDP - Hierarchical Drichlet Process\n\n> HDP, the Hierarchical Drichlet Process is an unsupervised topic model which figures out the number of topics on it's own.","metadata":{}},{"cell_type":"code","source":"hdp_model = HdpModel(corpus=corpus, id2word=dictionary)\nhdp_model.show_topics()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ LDA - Latent Dirchlet Allocation\n\n> LDA, or Latent Dirchlet Allocation is arguably the most famous topic modeling algorithm out there. Out here we create a simple topic model with 10 topics.","metadata":{}},{"cell_type":"code","source":"lda_model = LdaModel(corpus=corpus, num_topics=10, id2word=dictionary)\nlda_model.show_topics()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# 📊 pyLDAvis\n\nPython library for interactive topic model visualization. This is a port of the fabulous R package by Carson Sievert and Kenny Shirley.\n\n**pyLDAvis** is designed to help users interpret the topics in a topic model that has been fit to a corpus of text data. The package extracts information from a fitted LDA topic model to inform an interactive web-based visualization.\n\nThe visualization is intended to be used within an IPython notebook but can also be saved to a stand-alone HTML file for easy sharing.","metadata":{}},{"cell_type":"code","source":"import pyLDAvis.gensim\n\n\npyLDAvis.enable_notebook()\npyLDAvis.gensim.prepare(lda_model, corpus, dictionary)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> This is a great way to get a view of what words end up appearing in our documents, and what kind of document topics might be present.","metadata":{}}]}