| POS | DESCRIPTION | EXAMPLES |
|---|
\n \n| ADJ | adjective | **big, old, green, incomprehensible, first** |
\n| ADP | adposition | *in, to, during* |
\n| ADV | adverb | *very, tomorrow, down, where, there* |
\n| AUX | auxiliary | *is, has (done), will (do), should (do)* |
\n| CONJ | conjunction | *and, or, but* |
\n| CCONJ | coordinating conjunction | *and, or, but* |
\n| DET | determiner | *a, an, the* |
\n| INTJ | interjection | *psst, ouch, bravo, hello* |
\n| NOUN | noun | *girl, cat, tree, air, beauty* |
\n| NUM | numeral | *1, 2017, one, seventy-seven, IV, MMXIV* |
\n| PART | particle | *'s, not,* |
\n| PRON | pronoun | *I, you, he, she, myself, themselves, somebody* |
\n| PROPN | proper noun | *Mary, John, London, NATO, HBO* |
\n| PUNCT | punctuation | *., (, ), ?* |
\n| SCONJ | subordinating conjunction | *if, while, that* |
\n| SYM | symbol | *$, %, ยง, ยฉ, +, โ, ร, รท, =, :), ๐* |
\n| VERB | verb | *run, runs, running, eat, ate, eating* |
\n| X | other | *sfpksdpsxmsa* |
\n| SPACE | space |
\n\n***\n","metadata":{}},{"cell_type":"markdown","source":"___\n## Fine-grained Part-of-speech Tags\nTokens are subsequently given a fine-grained tag as determined by morphology:\n\n| POS | Description | Fine-grained Tag | Description | Morphology |
|---|
\n| ADJ | adjective | AFX | affix | Hyph=yes |
\n| ADJ | | JJ | adjective | Degree=pos |
\n| ADJ | | JJR | adjective, comparative | Degree=comp |
\n| ADJ | | JJS | adjective, superlative | Degree=sup |
\n| ADJ | | PDT | predeterminer | AdjType=pdt PronType=prn |
\n| ADJ | | PRP\\$ | pronoun, possessive | PronType=prs Poss=yes |
\n| ADJ | | WDT | wh-determiner | PronType=int rel |
\n| ADJ | | WP\\$ | wh-pronoun, possessive | Poss=yes PronType=int rel |
\n| ADP | adposition | IN | conjunction, subordinating or preposition | |
\n| ADV | adverb | EX | existential there | AdvType=ex |
\n| ADV | | RB | adverb | Degree=pos |
\n| ADV | | RBR | adverb, comparative | Degree=comp |
\n| ADV | | RBS | adverb, superlative | Degree=sup |
\n| ADV | | WRB | wh-adverb | PronType=int rel |
\n| CONJ | conjunction | CC | conjunction, coordinating | ConjType=coor |
\n| DET | determiner | DT | determiner | |
\n| INTJ | interjection | UH | interjection | |
\n| NOUN | noun | NN | noun, singular or mass | Number=sing |
\n| NOUN | | NNS | noun, plural | Number=plur |
\n| NOUN | | WP | wh-pronoun, personal | PronType=int rel |
\n| NUM | numeral | CD | cardinal number | NumType=card |
\n| PART | particle | POS | possessive ending | Poss=yes |
\n| PART | | RP | adverb, particle | |
\n| PART | | TO | infinitival to | PartType=inf VerbForm=inf |
\n| PRON | pronoun | PRP | pronoun, personal | PronType=prs |
\n| PROPN | proper noun | NNP | noun, proper singular | NounType=prop Number=sign |
\n| PROPN | | NNPS | noun, proper plural | NounType=prop Number=plur |
\n| PUNCT | punctuation | -LRB- | left round bracket | PunctType=brck PunctSide=ini |
\n| PUNCT | | -RRB- | right round bracket | PunctType=brck PunctSide=fin |
\n| PUNCT | | , | punctuation mark, comma | PunctType=comm |
\n| PUNCT | | : | punctuation mark, colon or ellipsis | |
\n| PUNCT | | . | punctuation mark, sentence closer | PunctType=peri |
\n| PUNCT | | '' | closing quotation mark | PunctType=quot PunctSide=fin |
\n| PUNCT | | \"\" | closing quotation mark | PunctType=quot PunctSide=fin |
\n| PUNCT | | `` | opening quotation mark | PunctType=quot PunctSide=ini |
\n| PUNCT | | HYPH | punctuation mark, hyphen | PunctType=dash |
\n| PUNCT | | LS | list item marker | NumType=ord |
\n| PUNCT | | NFP | superfluous punctuation | |
\n| SYM | symbol | # | symbol, number sign | SymType=numbersign |
\n| SYM | | \\$ | symbol, currency | SymType=currency |
\n| SYM | | SYM | symbol | |
\n| VERB | verb | BES | auxiliary \"be\" | |
\n| VERB | | HVS | forms of \"have\" | |
\n| VERB | | MD | verb, modal auxiliary | VerbType=mod |
\n| VERB | | VB | verb, base form | VerbForm=inf |
\n| VERB | | VBD | verb, past tense | VerbForm=fin Tense=past |
\n| VERB | | VBG | verb, gerund or present participle | VerbForm=part Tense=pres Aspect=prog |
\n| VERB | | VBN | verb, past participle | VerbForm=part Tense=past Aspect=perf |
\n| VERB | | VBP | verb, non-3rd person singular present | VerbForm=fin Tense=pres |
\n| VERB | | VBZ | verb, 3rd person singular present | VerbForm=fin Tense=pres Number=sing Person=3 |
\n| X | other | ADD | email | |
\n| X | | FW | foreign word | Foreign=yes |
\n| X | | GW | additional word in multi-word expression | |
\n| X | | XX | unknown | |
\n| SPACE | space | _SP | space | |
\n | | NIL | missing tag | |
\n
","metadata":{}},{"cell_type":"markdown","source":"## Working with POS Tags\n\nIn english language, the same string of characters can have different meanings, even within the same sentence. For this reason, morphology is important.","metadata":{}},{"cell_type":"code","source":"doc = nlp(\"I read books on NLP.\")\nr = doc[1]\nprint(f\"{r.text:{10}} {r.pos_:{8}} {r.tag_:{6}} {spacy.explain(r.tag_)}\\n\")\n\ndoc = nlp(\"I am reading a book on NLP.\")\nr = doc[2]\nprint(f\"{r.text:{10}} {r.pos_:{8}} {r.tag_:{6}} {spacy.explain(r.tag_)}\")","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:14:06.013675Z","iopub.execute_input":"2023-02-01T18:14:06.014138Z","iopub.status.idle":"2023-02-01T18:14:06.044244Z","shell.execute_reply.started":"2023-02-01T18:14:06.014099Z","shell.execute_reply":"2023-02-01T18:14:06.043065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Counting POS Tags\n\nThe `Doc.count_by()` method accepts a specific token attribute as its argument, and returns a frequency count of the given attribute as a dictionary object.","metadata":{}},{"cell_type":"code","source":"doc = nlp(\"The quick brown fox jumped over the lazy dog's back.\")\n\npos_count = doc.count_by(spacy.attrs.POS)\nprint(pos_count)\nnew = {}\nfor key, value in pos_count.items():\n new[doc.vocab[key].text] = value\n \nprint(new)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:14:09.806904Z","iopub.execute_input":"2023-02-01T18:14:09.807724Z","iopub.status.idle":"2023-02-01T18:14:09.828066Z","shell.execute_reply.started":"2023-02-01T18:14:09.807681Z","shell.execute_reply":"2023-02-01T18:14:09.826833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Gensim\n\nGensim is an open-source library for unsupervised topic modeling and natural language processing, using modern statistical machine learning. \n\n## Document \nIn Gensim, a document is an object of the text sequence type (commonly known as str in Python 3). A document could be anything from a short 140 character tweet, a single paragraph (i.e., journal article abstract), a news article, or a book.","metadata":{}},{"cell_type":"code","source":"!wget https://www.gutenberg.org/files/1342/1342-0.txt","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:14:13.525241Z","iopub.execute_input":"2023-02-01T18:14:13.525965Z","iopub.status.idle":"2023-02-01T18:14:19.047492Z","shell.execute_reply.started":"2023-02-01T18:14:13.525929Z","shell.execute_reply":"2023-02-01T18:14:19.046103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport spacy\n\nnlp = spacy.load('en_core_web_sm')\n\ndef read_file(file_name):\n with open(file_name, 'r') as file:\n return file.read()\n \ntext = read_file('1342-0.txt')\nprocessed_text = nlp(text)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:14:51.255773Z","iopub.execute_input":"2023-02-01T18:14:51.256528Z","iopub.status.idle":"2023-02-01T18:15:21.050331Z","shell.execute_reply.started":"2023-02-01T18:14:51.256476Z","shell.execute_reply":"2023-02-01T18:15:21.049034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Corpus\nA corpus is a collection of document objects. Corpora serve two roles in Gensim:\n1. Input for training a model.\n2. Documents to orgnize.","metadata":{}},{"cell_type":"code","source":"# An example of corpus, consists of 7045 sentences\nsentences = [s for s in processed_text.sents]\n\nprint(len(sentences))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:15:29.630973Z","iopub.execute_input":"2023-02-01T18:15:29.631391Z","iopub.status.idle":"2023-02-01T18:15:29.643947Z","shell.execute_reply.started":"2023-02-01T18:15:29.631360Z","shell.execute_reply":"2023-02-01T18:15:29.643145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(sentences[30:34])","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:15:31.047225Z","iopub.execute_input":"2023-02-01T18:15:31.047618Z","iopub.status.idle":"2023-02-01T18:15:31.054459Z","shell.execute_reply.started":"2023-02-01T18:15:31.047585Z","shell.execute_reply":"2023-02-01T18:15:31.053038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(processed_text.text.split())","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:15:33.140584Z","iopub.execute_input":"2023-02-01T18:15:33.141026Z","iopub.status.idle":"2023-02-01T18:15:33.263114Z","shell.execute_reply.started":"2023-02-01T18:15:33.140972Z","shell.execute_reply":"2023-02-01T18:15:33.261756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above example loads the entire corpus into memory. In practice, corpora may be very large, so loading them into memory may be impossible. Gensim intelligently handles such corpus by streaming one document at a time.","metadata":{}},{"cell_type":"markdown","source":"## Word2Vec\n\n\nUsing large amounts of unannotated plain text, word2vec learns relationships between words automatically. The output are vectors, one vector per word, with remarkable linear relationships.\n\nWord2Vec is very useful in automatic text tagging, recommender systems and machine translation.\n\n`Word2Vec`: is a more recent model that embeds words in a lower-dimentional vector space using a shallow neural network. The result is a set of word-vectors where vectors close together in vector space have similar meanings based on context, and word-vectors distant to each other have differing meanings. For example, `strong` and `powerful` would be close together and `strong` and `Paris` would be relatively far.","metadata":{}},{"cell_type":"code","source":"import gensim\nfrom gensim.models import Word2Vec\n\nprint(f\"Gensim Version: {gensim.__version__}\")\n\n# We need data for training the model\nprocessed_sentences = [sent.lemma_.split() for sent in processed_text.sents]\nprocessed_sentences[0]","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:15:35.345969Z","iopub.execute_input":"2023-02-01T18:15:35.346479Z","iopub.status.idle":"2023-02-01T18:15:35.715714Z","shell.execute_reply.started":"2023-02-01T18:15:35.346438Z","shell.execute_reply":"2023-02-01T18:15:35.714627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Word2Vec accepts several parameters that affect both training speed and quality\ninterchangeable_words_model = Word2Vec(\n sentences=processed_sentences,\n min_count=10, # Purning the internal dictionary\n# size=200, # the number of dimensions (N) gensim maps the word onto\n window=2, # \n compute_loss=True,\n sg=1\n)\n\n# print(len(interchangeable_words_model.wv.vocab))\n\n# getting the training loss\ntraining_loss = interchangeable_words_model.get_latest_training_loss()\nprint(f\"Training Loss: {training_loss}\")\n\nfor w, sim in interchangeable_words_model.wv.most_similar('Darcy'):\n print((w, sim))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:16:32.187673Z","iopub.execute_input":"2023-02-01T18:16:32.188569Z","iopub.status.idle":"2023-02-01T18:16:32.948024Z","shell.execute_reply.started":"2023-02-01T18:16:32.188518Z","shell.execute_reply":"2023-02-01T18:16:32.945829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluating\n\n`Word2Vec` training is an unsupervised task, there's no good way to objectively evaluate the result.","metadata":{}},{"cell_type":"markdown","source":"## FastText","metadata":{}},{"cell_type":"code","source":"# from gensim.models import FastText\n\n# model = FastText(window=2)\n# model.build_vocab(sentences=processed_sentences)\n# model.train(sentences=processed_sentences, total_examples=len(processed_sentences), epochs=10)\n\n# for w, sim in model.wv.most_similar('Darcy'):\n# print((w, sim))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:16:40.778295Z","iopub.execute_input":"2023-02-01T18:16:40.779553Z","iopub.status.idle":"2023-02-01T18:16:40.820768Z","shell.execute_reply.started":"2023-02-01T18:16:40.779497Z","shell.execute_reply":"2023-02-01T18:16:40.818756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# for w, sim in model.wv.most_similar('Darcy', topn=20):\n# print((w, sim))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:11:49.515488Z","iopub.status.idle":"2023-02-01T18:11:49.515934Z","shell.execute_reply.started":"2023-02-01T18:11:49.515729Z","shell.execute_reply":"2023-02-01T18:11:49.515750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model = FastText(window=50)\n# model.build_vocab(sentences=processed_sentences)\n# model.train(sentences=processed_sentences, total_examples=len(processed_sentences), epochs=10)\n\n# for w, sim in model.wv.most_similar('Darcy'):\n# print((w, sim))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:11:49.517121Z","iopub.status.idle":"2023-02-01T18:11:49.517514Z","shell.execute_reply.started":"2023-02-01T18:11:49.517323Z","shell.execute_reply":"2023-02-01T18:11:49.517343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(\"night\" in model.wv.vocab)\n# print(\"nights\" in model.wv.vocab)","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:11:49.518516Z","iopub.status.idle":"2023-02-01T18:11:49.518914Z","shell.execute_reply.started":"2023-02-01T18:11:49.518721Z","shell.execute_reply":"2023-02-01T18:11:49.518741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(model.wv.similarity(\"nights\", \"night\"))\n# print(model.wv.similarity(\"tonight\", \"night\"))","metadata":{"execution":{"iopub.status.busy":"2023-02-01T18:11:49.520105Z","iopub.status.idle":"2023-02-01T18:11:49.520496Z","shell.execute_reply.started":"2023-02-01T18:11:49.520305Z","shell.execute_reply":"2023-02-01T18:11:49.520325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}