{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# BMS Molecular Translation - Train InChI Tokenizer\n\n## 1. Introduction\n\nRecently, many deep learning NLP models use subword tokenization (e.g. [BPE](https://arxiv.org/abs/1508.07909v5), [WordPiece](https://arxiv.org/pdf/1609.08144v2.pdf)) rather than word-level or character-level tokenizations.\nSubword tokenization can efficiently reduce the vocabulary size to the desired scale, and ensures to learn deeper semantics than the character-level tokenization.\nDue to the reasons, subword tokenization is adopted to various tasks like translation, question-answering, reading comprehension and text generation.\n\nIn [this competition](https://www.kaggle.com/c/bms-molecular-translation/code), we need to train a model which generates **InChI** text by attending from old chemical image.\nSince generating InChI sequences is identical to the NLP's one, we can consider the subword tokenization to **InChI** strings.\nIn this notebook, we are going to train the subword tokenizer for **InChI** format and check out a distribution of length of the tokenized sequences to determine the proper maximum sequence length.\n\n## 2. Train Tokenizer\n\nFirst, install [`tokenizers`](https://huggingface.co/docs/tokenizers/python/latest/).\n`tokenizers` is a library containing today's most used tokenizers mentioned above.\nIt provides an implementation of those tokenizers and an interface for training, tokenizing, and pipelining the entire encoding procedures.","metadata":{}},{"cell_type":"code","source":"!pip install -qq -U allennlp transformers tokenizers","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After installing the library, load the necessary modules.","metadata":{}},{"cell_type":"code","source":"import tqdm\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Punctuation\nfrom tokenizers.processors import TemplateProcessing\nfrom tokenizers.decoders import WordPiece as WordPieceDecoder","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's train our **InChI** tokenizer.\nAs you can see, `train_labels.csv` contains image ids and **InChI** strings.","metadata":{}},{"cell_type":"code","source":"samples = pd.read_csv('../input/bms-molecular-translation/train_labels.csv')\nsamples.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The below tokens, which do not appear to the target texts, will be added to the vocabulary.\n- `[UNK]`: Unknown token. It is used when the word cannot replaced to subword combination.\n- `[BOS]`: Begin-of-sequence token. It is added to the front of every sequences. You can feed this token to generate sequences without previous contexts.\n- `[EOS]`: End-of-sequence token. It announce that the sequence is ended and the tokens after this are meaningless.\n- `[PAD]`: Padding token. It is added to match the sequence length with each other in same batch.\n\nThe below code will train the tokenizer by constructing vocabulary with 256 subword tokens, including the above special tokens.\nFrequently appeard subword pairs will be merged and added to the vocabulary.\nIt is repeated until the vocabulary is filled.","metadata":{}},{"cell_type":"code","source":"tokenizer = Tokenizer(WordPiece(unk_token='[UNK]'))\ntokenizer.pre_tokenizer = Punctuation()\n\ntrainer = WordPieceTrainer(\n    vocab_size=256, \n    min_frequency=2,\n    special_tokens=['[UNK]', '[BOS]', '[EOS]', '[PAD]']\n)\ntokenizer.train_from_iterator(samples['InChI'], trainer=trainer)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's all! We've train our own **InChI** tokenizer successfully. You can tokenize **InChI** strings to subword tokens through this tokenizer.","metadata":{}},{"cell_type":"markdown","source":"## 3. Visualize Sequence Lengths\n\nNow we are wondering the range of sequence length.\nIt is important to restrict the maximum sequence length.\nWhich `max_seq_len` is proper?\n\nTo decide `max_seq_len`, let's visualize the distribution of `seq_len`s.\nFirst of all, configure the encoding template.\nAs I mentioned above, `[BOS]` and `[EOS]` tokens will be added before and after the sequences respectively.\nWe can pipelining this post-processing by using `TemplateProcessing` class.","metadata":{}},{"cell_type":"code","source":"tokenizer.post_processor = TemplateProcessing(\n    single=\"[BOS] $A [EOS]\",\n    special_tokens=[\n        (\"[BOS]\", tokenizer.token_to_id(\"[BOS]\")),\n        (\"[EOS]\", tokenizer.token_to_id(\"[EOS]\")),\n    ],\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The belows are some examples of tokenized **InChI** texts.\nAll the tokens are mapped to their indices and passed to the models.","metadata":{}},{"cell_type":"code","source":"print(' '.join(tokenizer.encode(samples.iloc[80000, 1]).tokens))\nprint(' '.join(tokenizer.encode(samples.iloc[53242, 1]).tokens))\nprint(' '.join(tokenizer.encode(samples.iloc[45212, 1]).tokens))\nprint(' '.join(tokenizer.encode(samples.iloc[782120, 1]).tokens))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using the trained tokenizer and encoding template, let's tokenize all **InChI** sequences and plot the histogram of their lengths.","metadata":{}},{"cell_type":"code","source":"lengths = []\nfor inchi in tqdm.tqdm(samples['InChI']):\n    lengths.append(len(tokenizer.encode(inchi).ids))","metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(max(lengths))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nplt.hist(lengths, bins=500)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! It seems that `256` is the proper `max_seq_len`!\nLet's configure the decoding, padding and truncation settings to the tokenizer.","metadata":{}},{"cell_type":"code","source":"tokenizer.decoder = WordPieceDecoder()\ntokenizer.enable_padding(pad_id=tokenizer.token_to_id(\"[PAD]\"), pad_token='[PAD]', pad_to_multiple_of=8)\ntokenizer.enable_truncation(max_length=256)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Save Tokenizer\n\nSo, how can we use this tokenizer? The answer is simple.\nWe're going to save this tokenizer to `tokenizer.json`.\nYou can use the trained tokenizer anytime, by simply loading the `tokenizer.json` file.","metadata":{}},{"cell_type":"code","source":"tokenizer.save('tokenizer.json')","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}