{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":16295,"databundleVersionId":1099992,"sourceType":"competition"},{"sourceId":17777,"databundleVersionId":869809,"sourceType":"competition"},{"sourceId":19018,"databundleVersionId":1053282,"sourceType":"competition"},{"sourceId":21733,"databundleVersionId":1408234,"sourceType":"competition"},{"sourceId":1863123,"sourceType":"datasetVersion","datasetId":1108793},{"sourceId":2380587,"sourceType":"datasetVersion","datasetId":1167113}],"dockerImageVersionId":30066,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📌 Notebook Goals\n> - Learn how to use the power of `spaCy` to clean textual data.\n> - Use different Topic Modelling techniques like `LDA (Latent Dirichlet Allocation)`, `LSI (Latent Semantic Indexing)`, and `HDP (Hierarchical Drichlet Process)`\n---\n\n# 📚 Topic Modelling Overview\nLet's understand the general concept of Topic Modelling and why it's important! \n> - Topic Modeling allows for us to efficiently analyze large volumes of text by clustering documents into topics.\n> - A large amount of text data is unlabeled meaning we can't apply supervised learning approaches to create machine learning models for the data! In this case of text data, this means attempting to discover clusters of documents, grouped together by topic.\n> - A very important idea to keep in mind here is that we don't know the correct topic or right answer! All we know is that the documents clustered together share similar topic ideas. It is up to us to identify what these topics represent.\n\n---\n# 📑 Text Analysis Tutorial\n\n> Our steps, naturally, is setting up ouy imports. We will be using spaCy for data pre-processing and computational linguistics, gensim for topic modelling, scikit-learn for classification, and Keras for text generation. ","metadata":{}},{"cell_type":"markdown","source":"# ✔️ Import Libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\n\nimport spacy \nfrom spacy import displacy\n\nimport gensim\nfrom gensim.corpora import Dictionary\nfrom gensim.models import LdaModel, CoherenceModel, LsiModel, HdpModel\nfrom gensim.models.wrappers import LdaMallet\n\nimport matplotlib.pyplot as plt\nimport sklearn\nimport keras\n\nimport warnings\n\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\n# warnings.simplefilter('once')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# 📂 Gathering Data\n\n> The dataset we will be working with will be the Lee corpus which is a shortened of the Lee Background Corpus, and the 20NG dataset. ","metadata":{}},{"cell_type":"code","source":"test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])\nprint(test_data_dir)\nlee_train_file = test_data_dir + os.sep + 'lee_background.cor'\nprint(lee_train_file)\ntext = open(lee_train_file).read()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# 🧹 Cleaning Data\n\n> We can't have state-of-the-art results without data which is as good. Let's spend this section working on cleaning and understanding our data set. We will be checking out `spaCy`, an industry grade text-processing package.","metadata":{}},{"cell_type":"code","source":"nlp = spacy.load('en')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> For safe measure, let's add some stopwords. It's a newspaper corpus, so it is likely we will be coming across variations of 'said', 'Mister', and 'Mr'... which will not really add any value to the topic models.","metadata":{}},{"cell_type":"code","source":"my_stop_words = ['say', '\\s', 'mr', 'Mr', 'said', 'says', 'saying', 'today', 'be']\nfor stopword in my_stop_words:\n    lexeme = nlp.vocab[stopword]\n    lexeme.is_stop = True","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"doc = nlp(text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# doc","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💹 Computational Linguistics\n\nNow that we have our doc object. We can see that the doc object now contains the entire corpus. This is important because we will be using the doc object to create our corpus for the machine learning algorithms. When creating a corpus for `gensim/scikit-learn`, we sometimes forget the incredible power which `spaCy` packs in its pipeline, so we will briefly demonstrate the same in this section with a smaller example sentence.\n","metadata":{}},{"cell_type":"code","source":"sent = nlp('Last Thursday, Manchester United defeated AC Milan at San Siro.')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🔖 POS-Tagging","metadata":{}},{"cell_type":"code","source":"for token in sent:\n    print(token.text, token.pos_, token.tag_)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🔖 NER-Tagging","metadata":{}},{"cell_type":"code","source":"for token in sent:\n    print(token.text, token.ent_type_)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for ent in sent.ents:\n    print(ent.text, ent.label_)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"displacy.render(sent, style='ent', jupyter=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧮 Dependency Parsing","metadata":{}},{"cell_type":"code","source":"for chunk in sent.noun_chunks:\n    print(chunk.text, chunk.root.text, chunk.root.dep_, chunk.root.head.text)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for token in sent:\n    print(token.text, token.dep_, token.head.text, token.head.pos_,\n         [child for child in token.children])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"displacy.render(sent, style='dep', jupyter=True, options={'distance':90})","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 🧹 Continuing Cleaning\n\n> Have a quick look at the output of the doc object. It seems like nothing, right? But spaCy's internal data structure has done all the work for us. Let's see how we can create our corpus.","metadata":{}},{"cell_type":"code","source":"# We add some words to the stop word list\ntexts, article = [], []\n\nfor word in doc:\n    \n    if word.text != '\\n' and not word.is_stop and not word.is_punct and not word.like_num and word.text != 'I':\n        article.append(word.lemma_)\n        \n    if word.text == '\\n':\n        texts.append(article)\n        article = []","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(texts[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> - And this is the magic of spaCy - just like that, we've managed to get rid of stopwords, puctuation markers, and added lemmatized word.\n> - Sometimes topic modeling make more sense when `New` and `York` are treated as `New York` - we can do this by creating a bigram model and modifying our corpus accordingly.","metadata":{}},{"cell_type":"code","source":"bigram = gensim.models.phrases.Phrases(texts)\ntexts = [bigram[line] for line in texts]\ntexts = [bigram[line] for line in texts]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(texts[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dictionary = Dictionary(texts)\ncorpus = [dictionary.doc2bow(text) for text in texts]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(corpus[1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Now we are done with a very important part of text analysis - the data cleaning and setting up of corpus. It must kept in mind that we created the corpus the way we did because that's how gensim requires it - most algorithms still require one to clean the data set the way we did, by removing stop words and numbers, adding the lemmatized form of the word, and using bigrams.","metadata":{}},{"cell_type":"markdown","source":"---\n# 📚 Topic Modeling\n\n> Topic Modeling refers to the probabilistic modeling of text document as topics. Gensim remains the most popular library to perform such modelling, and we will be using it to perform our topic modelling.","metadata":{}},{"cell_type":"markdown","source":"## ✔️ LSI - Latent Semantic Indexing\n\n> LSI stands for Latent Semantic Indexing - It is a popular information retreival method which works by decomposing the original matrix of words to maintain key topics. ","metadata":{}},{"cell_type":"code","source":"lsi_model = LsiModel(corpus=corpus, num_topics=10, id2word=dictionary)\nlsi_model.show_topics(num_topics=5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ HDP - Hierarchical Drichlet Process\n\n> HDP, the Hierarchical Drichlet Process is an unsupervised topic model which figures out the number of topics on it's own.","metadata":{}},{"cell_type":"code","source":"hdp_model = HdpModel(corpus=corpus, id2word=dictionary)\nhdp_model.show_topics()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ LDA - Latent Dirchlet Allocation\n\n> LDA, or Latent Dirchlet Allocation is arguably the most famous topic modeling algorithm out there. Out here we create a simple topic model with 10 topics.","metadata":{}},{"cell_type":"code","source":"lda_model = LdaModel(corpus=corpus, num_topics=10, id2word=dictionary)\nlda_model.show_topics()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# 📊 pyLDAvis","metadata":{}},{"cell_type":"code","source":"import pyLDAvis.gensim","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pyLDAvis.enable_notebook()\npyLDAvis.gensim.prepare(lda_model, corpus, dictionary)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> This is a great way to get a view of what words end up appearing in our documents, and what kind of document topics might be present.","metadata":{}}]}