{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.core.display import HTML\ndef css_styling():\n styles = open(\"../input/titlestyle/style2.css\", \"r\").read()\n return HTML(styles)\ncss_styling()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-08-30T13:05:47.998942Z","iopub.execute_input":"2021-08-30T13:05:47.999326Z","iopub.status.idle":"2021-08-30T13:05:48.029436Z","shell.execute_reply.started":"2021-08-30T13:05:47.999243Z","shell.execute_reply":"2021-08-30T13:05:48.028394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

Content

\n
\n\n* [Named-Entity Recognition (NER) - Intro and Concept](#topic1)\n* [Applications](#topic2)\n* [POS Tagging](#topic3)\n* [Chunking](#topic4)\n* [IOB Tags](#topic5)\n* [Extracting Named Entities (NLTK)](#topic6)\n* [spaCy Named Entity Recognition](#topic7)\n* [Visualizing Named Entities (spaCy)](#topic8)\n* [NER BERT Hugging Face](#topic9)\n* [NER BERT Finetuning Existing Model](#topic10)\n* [Other Usefull Resources and References](#topic11)","metadata":{}},{"cell_type":"markdown","source":" \n
\n

Named-Entity Recognition (NER)

\n
\n
\n\nNamed-entity recognition (NER) is a problem that has a goal to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, etc.\n

\nNER is used in many fields in NPL and it can help to answer some questions such as:\n\n\nState-of-the-art NER systems for English produce near-human performance where the best system scored 93.39% of F-measure while human annotators have a score around 97%.\n

\nNamed-entity recognition is often broken down into two distinct problems:\n\nDetection of names is typical simplified to a segmentation problem where a single name might be constructed of several substrings. For example \"Bank of America\" is a single name despite that the substring \"America\" is itself a name. \nClassification of names requires choosing an ontology by which to organize categories of things.\n

\nWhile doing NER, besides the correct and incorrect predicted terms, we'll probably face some \"partially correct\" predictions. For example:\n
\n\n

\n
","metadata":{}},{"cell_type":"markdown","source":" \n
\n

Applications

\n
\n
\n\nNER can be appllied in many real-world situations when analyzing a large quantity of text is helpful. Some examples of NER includes:\n\n\n \n
","metadata":{}},{"cell_type":"markdown","source":" \n
\n

POS Tagging

\n
\n
\n\nPOS Tagging (Parts of Speech Tagging) is a process to mark up the words in text format for a particular part of a speech based on its definition and context. It is responsible for text reading in a language and assigning some specific token (Parts of Speech) to each word. It is also called grammatical tagging.\n
","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nfrom nltk import word_tokenize, pos_tag\n\n#os.listdir('../input/nlp-getting-started/train.csv')\n# read data from nlp-getting-started\nnlp_start_df = pd.read_csv('../input/nlp-getting-started/train.csv')\n\n# take one example sentence \nex = nlp_start_df.loc[159]['text']\n\n# tokenize the sencence and apply POS tagging\nsent = pos_tag(word_tokenize(ex))\nsent","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nFrom the example above, we have that the word 'Experts' is 'NNS' (noun plural), word 'France' is 'NNP' (proper noun) and 'French' is 'JJ' (adjective). The whole list of abbreviations can be found here. After this step, we can start with noun phrase chunking to named entities\n
\n \n
\n

Chunking

\n
\n
\n\nChunking in NLP is a process of grouping small pieces of information into large units. The primary use of Chunking is making groups of \"noun phrases.\" It is used to add structure to the sentence by following POS tagging combined with regular expressions. The resulted group of words are called \"chunks.\"\n\nThere are no pre-defined rules for Chunking, but we can made according to our needs. Thus, if we want to chunk only 'NN' tags, we need to use pattern 
`mychunk:{<NN>}`
but if we need to chunk all types of tags which start with 'NN', we'll use 
`mychunk:{<NN.*>}`.
More about regex patterns can be found here\n
","metadata":{}},{"cell_type":"code","source":"!pip install svgling","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from nltk import RegexpParser\nfrom nltk.draw.tree import TreeView\nfrom IPython.display import Image\nimport svgling\n\n# chunk all adjacence nouns\npatterns= \"\"\"mychunk:{+}\"\"\"\nchunker = RegexpParser(patterns)\noutput = chunker.parse(sent)\nprint(\"After Chunking\",output)\nsvgling.draw_tree(output)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

IOB Tags

\n
\n
\n\nSimilarly as part-of-speech tags, IOB tags are a slightly different way for representing chunk structures. This format can denote the inside, outside, and beginning of a chunk.\n
","metadata":{}},{"cell_type":"code","source":"from nltk.chunk import conlltags2tree, tree2conlltags\nfrom pprint import pprint\n\niob_tagged = tree2conlltags(output)\niob_tagged","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

Extracting Named Entities

\n
\n
\nRecognizing a named entity is a specific kind of chunk extraction that uses entity tags along with chunk tags. Common entity tags include PERSON, LOCATION, and ORGANIZATION. NLTK has already a pre-trained named entity chunker which can be used using ne_chunk() method in the nltk.chunk module.\n
","metadata":{}},{"cell_type":"code","source":"from nltk.chunk import ne_chunk\n\ndef extract_ne(trees, labels):\n \n ne_list = []\n for tree in ne_res:\n if hasattr(tree, 'label'):\n if tree.label() in labels:\n ne_list.append(tree)\n \n return ne_list\n \nne_res = ne_chunk(pos_tag(word_tokenize(ex)))\nlabels = ['ORGANIZATION']\n\nprint(extract_ne(ne_res, labels))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

spaCy Named Entity Recognition

\n
\n
\nspaCy features an extremely fast statistical entity recognition system, that assigns labels to contiguous spans of tokens. The default trained pipelines can identify a variety of named and numeric entities, including companies, locations, organizations and products.\n
","metadata":{}},{"cell_type":"code","source":"import sqlite3\n\ncnx = sqlite3.connect('../input/wikibooks-dataset/wikibooks.sqlite')\ndf_wikibooks = pd.read_sql_query(\"SELECT * FROM en\", cnx)\ndf_wikibooks.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import spacy\n\nnlp = spacy.load(\"en_core_web_sm\")\nwiki_ex = df_wikibooks.iloc[11]['body_text']\ndoc = nlp(wiki_ex)\ndoc","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('All entity types that spacy recognised from the document above')\nset([ent.label_ for ent in doc.ents])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Persons from the document above')\nprint(set([ent for ent in doc.ents if ent.label_ == 'PERSON']))\nprint('Organizations from the document above')\nprint(set([ent for ent in doc.ents if ent.label_ == 'ORG']))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

Visualizing Named Entities

\n
","metadata":{}},{"cell_type":"code","source":"from spacy import displacy\ndisplacy.render(doc, style=\"ent\", jupyter=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

NER BERT Hugging Face

\n
\n
\nBERT (Bidirectional Encoder Representations from Transformers) is a neural network that is capable of parsing language in the same way a human does. It uses word embeddings to translate words into numbers and then back again, allowing it to understand word context and meaning.\n

\nBERT focuses on understanding language comprehension, identifying linguistic cues in the text, and determining the context in which words are used.\n

\nBesides text classification, summarization, question answering and several other tasks, BERT can be used to solve NER.\n

\nHugging face is one of the most used NLP libraries. With this library, we can leverage popular NLP models, such as BERT, DistilBERT roBERTa and use those models to manipulate text in one way or another. Hugging Face provides simple access to a variety of models and datasets used for all possible NLP tasks.\n

\nModels for NER tasks can be found under the \"Token Classification\" section here https://huggingface.co/models?pipeline_tag=token-classification&sort=downloads. We can use the default model or select one from the mentioned link.\n

\nGreat Hugging Face course about transformers and the usage of the library can be found here https://huggingface.co/course/chapter0?fw=pt\n
","metadata":{}},{"cell_type":"code","source":"from transformers import pipeline\n\nnlp_start_df = pd.read_csv('../input/nlp-getting-started/train.csv')\n# take one example sentence \nex = nlp_start_df.loc[159]['text']\n\ngenerator = pipeline(\"ner\",\n #model=\"dslim/bert-base-NER\",\n grouped_entities=True)\ngenerator(ex)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

NER BERT Finetuning Existing Model

\n
\n
\n\nThe process of training a neural network is a difficult and time-consuming process and for most of the users not even feasible. Because of that, instead of training the model from scratch, we can use models from Hugging Face which has been trained using a large amount of text. \n

\nThese types of models through training developed a statistical understanding of the language they have been trained on, but they might not be useful for our specific task. In order to utilize the knowledge of the model, we can apply fine-tuning. It means that we can take pretrained model and train it a little bit more with our annotated data.\n

\nThis process is called transfer learning when the knowledge is transfered from one model to another one and that strategy is often used in deep learning.\n

\nFirst of all, we can show the annotated data for the NER task.\n
","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndata_dir = '../input/entity-annotated-corpus'\ndf = pd.read_csv(f'{data_dir}/ner_dataset.csv')\ndf['Sentence #'] = df['Sentence #'].ffill()\ndf_gr = df.groupby('Sentence #').agg(lambda x: list(x))\n\ndf_gr.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-29T12:27:47.182599Z","iopub.execute_input":"2021-08-29T12:27:47.182975Z","iopub.status.idle":"2021-08-29T12:27:55.195321Z","shell.execute_reply.started":"2021-08-29T12:27:47.182946Z","shell.execute_reply":"2021-08-29T12:27:55.194275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\nIn our case, we'll need columns `Word` (tokenized sentence) and `Tag` (entities in the sentence). Also, in order to fine-tune the model, we'll need to have the same entities as the model is trained on. Let's print entities from our data set and from the pretrained model.\n
","metadata":{}},{"cell_type":"code","source":"tags = []\nfor tag in df_gr['Tag'].to_list():\n tags.extend(tag)\nprint('Entities in our data set')\nset(tags)","metadata":{"execution":{"iopub.status.busy":"2021-08-29T12:27:55.198813Z","iopub.execute_input":"2021-08-29T12:27:55.199126Z","iopub.status.idle":"2021-08-29T12:27:55.235901Z","shell.execute_reply.started":"2021-08-29T12:27:55.199098Z","shell.execute_reply":"2021-08-29T12:27:55.233481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import AutoModelForTokenClassification\n\nmodel_checkpoint = \"dslim/bert-base-NER\"\nmodel = AutoModelForTokenClassification.from_pretrained(model_checkpoint)\n\nprint('Entities from the pretrained model')\nmodel.config.id2label","metadata":{"execution":{"iopub.status.busy":"2021-08-29T12:27:55.238874Z","iopub.execute_input":"2021-08-29T12:27:55.239565Z","iopub.status.idle":"2021-08-29T12:28:22.652044Z","shell.execute_reply.started":"2021-08-29T12:27:55.239522Z","shell.execute_reply":"2021-08-29T12:28:22.650979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nCreate the mapping for converting entities from our data set into the entity id (key) from the 0","metadata":{}},{"cell_type":"code","source":"entity_mapping = {\n'O':0,'B-per':3, 'I-per':4, 'B-org':5, 'I-org':6,'B-geo':7, 'I-geo':8,\n'B-art':1, 'B-eve':1 , 'B-gpe':1, 'B-nat':1, 'B-tim':1,\n'I-art':1, 'I-eve':1 , 'I-gpe':1, 'I-nat':1, 'I-tim':1,\n}","metadata":{"execution":{"iopub.status.busy":"2021-08-29T12:28:22.654162Z","iopub.execute_input":"2021-08-29T12:28:22.654888Z","iopub.status.idle":"2021-08-29T12:28:22.661987Z","shell.execute_reply.started":"2021-08-29T12:28:22.654839Z","shell.execute_reply":"2021-08-29T12:28:22.660196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class NERDataset:\n def __init__(self, df):\n # input is annotated data frame\n self.texts = df['Word'].to_list()\n self.tags = df['Tag'].to_list()\n \n def __len__(self):\n return len(self.texts)\n \n def __getitem__(self, item):\n text = self.texts[item]\n tags = self.tags[item]\n \n ids = []\n target_tag =[]\n \n # tokenize words and define tags accordingly\n # running -> [run, ##ning]\n # tags - ['O', 'O']\n for i, s in enumerate(text):\n inputs = tokenizer.encode(s, add_special_tokens=False)\n input_len = len(inputs)\n ids.extend(inputs)\n target_tag.extend([entity_mapping[tags[i]]] * input_len)\n \n # truncate\n ids = ids[:MAX_LEN - 2]\n target_tag = target_tag[:MAX_LEN - 2]\n \n # add special tokens\n ids = [101] + ids + [102]\n target_tag = [0] + target_tag + [0]\n mask = [1] * len(ids)\n token_type_ids = [0] * len(ids)\n \n # construct padding\n padding_len = MAX_LEN - len(ids)\n ids = ids + ([0] * padding_len)\n mask = mask + ([0] * padding_len)\n token_type_ids = token_type_ids + ([0] * padding_len)\n target_tag = target_tag + ([0] * padding_len)\n \n return {'input_ids': torch.tensor(ids, dtype=torch.long),\n 'attention_mask': torch.tensor(mask, dtype=torch.long),\n 'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),\n 'labels': torch.tensor(target_tag, dtype=torch.long)\n }","metadata":{"execution":{"iopub.status.busy":"2021-08-29T13:10:34.431407Z","iopub.execute_input":"2021-08-29T13:10:34.431756Z","iopub.status.idle":"2021-08-29T13:10:34.447684Z","shell.execute_reply.started":"2021-08-29T13:10:34.431726Z","shell.execute_reply":"2021-08-29T13:10:34.446568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom transformers import AutoTokenizer\nimport torch\n\ndf_train, df_val = train_test_split(df_gr, test_size=0.2, random_state=42)\ndf_val, df_test = train_test_split(df_val, test_size=0.5, random_state=42)\n\nmodel_checkpoint = \"dslim/bert-base-NER\"\ntokenizer = AutoTokenizer.from_pretrained(model_checkpoint)\n\nMAX_LEN = 128\n\ndata_train = NERDataset(df_train)\ndata_val = NERDataset(df_val)\ndata_test = NERDataset(df_test)\n\n# initialize DataLoader used to return batches for training/validation\nloader_train = torch.utils.data.DataLoader(\n data_train, batch_size=32, num_workers=4\n)\n\nloader_val = torch.utils.data.DataLoader(\n data_val, batch_size=32, num_workers=4\n)","metadata":{"execution":{"iopub.status.busy":"2021-08-29T13:10:48.9071Z","iopub.execute_input":"2021-08-29T13:10:48.907516Z","iopub.status.idle":"2021-08-29T13:10:51.637694Z","shell.execute_reply.started":"2021-08-29T13:10:48.907483Z","shell.execute_reply":"2021-08-29T13:10:51.636587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from transformers import AdamW, AutoModelForTokenClassification, get_scheduler\nfrom tqdm.notebook import tqdm\nfrom sklearn.metrics import f1_score, accuracy_score\n\nmodel = AutoModelForTokenClassification.from_pretrained(model_checkpoint)\n\n# just train the linear classifier on top of BERT\nparam_optimizer = list(model.classifier.named_parameters())\noptimizer_grouped_parameters = [{\"params\": [p for n, p in param_optimizer]}]\noptimizer = AdamW(\n optimizer_grouped_parameters,\n lr=3e-5,\n eps=1e-12\n)\n## full finetuning\n#optimizer = AdamW(model.parameters())\n\ndevice = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\nmodel.to(device)\n\n# add scheduler to linearly reduce the learning rate throughout the epochs.\nnum_epochs = 3\nnum_training_steps = num_epochs * len(loader_train)\nlr_scheduler = get_scheduler(\n \"linear\",\n optimizer=optimizer,\n num_warmup_steps=0,\n num_training_steps=num_training_steps\n)\n\n\nprogress_bar = tqdm(range(num_training_steps))\nfor epoch in range(num_epochs):\n model.train()\n final_loss = 0\n predictions , true_labels = [], []\n for batch in loader_train:\n batch = {k: v.to(device) for k, v in batch.items()}\n outputs = model(**batch)\n loss = outputs.loss\n loss.backward()\n true_labels.extend(batch['labels'].detach().cpu().numpy().ravel())\n predictions.extend(np.argmax(outputs[1].detach().cpu().numpy(), axis=2).ravel())\n \n optimizer.step()\n lr_scheduler.step()\n optimizer.zero_grad()\n progress_bar.update(1)\n final_loss+=loss.item()\n \n print(f'Training loss: {final_loss/len(loader_train)}')\n print('Training F1: {}'.format(f1_score(predictions, true_labels, average='macro')))\n print(f'Training acc: {accuracy_score(predictions, true_labels)}')\n print('*'*20)\n \n model.eval()\n final_loss = 0\n predictions , true_labels = [], []\n for batch in loader_val:\n batch = {k: v.to(device) for k, v in batch.items()}\n outputs = model(**batch)\n final_loss+=outputs.loss.item()\n true_labels.extend(batch['labels'].detach().cpu().numpy().ravel())\n predictions.extend(np.argmax(outputs[1].detach().cpu().numpy(), axis=2).ravel())\n print(f'Validation loss: {final_loss/len(loader_val)}')\n print('Vallidation F1: {}'.format(f1_score(predictions, true_labels, average='macro')))\n print(f'Validaton acc: {accuracy_score(predictions, true_labels)}')\n print('*'*20)","metadata":{"execution":{"iopub.status.busy":"2021-08-29T13:11:16.490786Z","iopub.execute_input":"2021-08-29T13:11:16.49119Z","iopub.status.idle":"2021-08-29T13:35:26.859384Z","shell.execute_reply.started":"2021-08-29T13:11:16.491144Z","shell.execute_reply":"2021-08-29T13:35:26.858196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\n\n# test the model\ntest_sentence = \"\"\"\nMr. Trump’s tweets began just moments after a Fox News report by Mike Tobin, a \nreporter for the network, about protests in Minnesota and elsewhere. \n\"\"\"\ntokenized_sentence = tokenizer.encode(test_sentence)\ninput_ids = torch.tensor([tokenized_sentence]).cuda()\nwith torch.no_grad():\n output = model(input_ids)\nlabel_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)\n# join bpe split tokens\ntokens = tokenizer.convert_ids_to_tokens(input_ids.to('cpu').numpy()[0])\nnew_tokens, new_labels = [], []\n\nfor token, label_idx in zip(tokens, label_indices[0]):\n if token.startswith(\"##\"):\n new_tokens[-1] = new_tokens[-1] + token[2:]\n else:\n new_labels.append(label_idx)\n new_tokens.append(token)\n \nfor token, label in zip(new_tokens, new_labels):\n print(\"{}\\t{}\".format(model.config.id2label[label], token))","metadata":{"execution":{"iopub.status.busy":"2021-08-29T13:35:26.86169Z","iopub.execute_input":"2021-08-29T13:35:26.862171Z","iopub.status.idle":"2021-08-29T13:35:26.904056Z","shell.execute_reply.started":"2021-08-29T13:35:26.86211Z","shell.execute_reply":"2021-08-29T13:35:26.90237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n
\n

Other Usefull Resources and References

\n
","metadata":{}},{"cell_type":"markdown","source":"* https://en.wikipedia.org/wiki/Named-entity_recognition\n* https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da\n* https://www.guru99.com/pos-tagging-chunking-nltk.html\n* https://www.geeksforgeeks.org/nlp-extracting-named-entities/\n* https://spacy.io/usage/linguistic-features#named-entities\n* https://www.kaggle.com/abhishek/entity-extraction-model-using-bert-pytorch\n* https://huggingface.co/course/chapter3/4?fw=pt\n* https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/\n* https://www.lighttag.io/blog/sequence-labeling-with-transformers/example\n* https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb#scrollTo=545PP3o8IrJV\n* https://discuss.huggingface.co/search?q=token%20classification","metadata":{}}]}