{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NLP Albumentations\n\nHi everyone!\n\nRecently I have published my [inference kernel](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta)\n\nNow I would like to share with you, my friends, experience in computer vision competition!\n\nCV? Yes, DL/CV/NLP are very similar.\n\nI have got good boost when I used this great library [albumentations](https://github.com/albumentations-team/albumentations) \n\n![](https://camo.githubusercontent.com/fd2405ab170ab4739c029d7251f5f7b4fac3b41c/68747470733a2f2f686162726173746f726167652e6f72672f776562742f62642f6e652f72762f62646e6572763563746b75646d73617a6e687734637273646669772e6a706567)"},{"metadata":{},"cell_type":"markdown","source":"## MAIN IDEA\n\nIn this competitions I needed similar NLP tool for creating nice training pipeline with augmentations for texts.\n\nSo I started searching another lib, but finally I decided create some similar classes for using [albumentations](https://github.com/albumentations-team/albumentations) for text.\n\nSo, let's start!"},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nimport re\nimport pandas as pd\nfrom nltk import sent_tokenize\nfrom tqdm import tqdm\nfrom albumentations.core.transforms_interface import DualTransform, BasicTransform","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class NLPTransform(BasicTransform):\n    \"\"\" Transform for nlp task.\"\"\"\n    LANGS = {\n        'en': 'english',\n        'it': 'italian', \n        'fr': 'french', \n        'es': 'spanish',\n        'tr': 'turkish', \n        'ru': 'russian',\n        'pt': 'portuguese'\n    }\n\n    @property\n    def targets(self):\n        return {\"data\": self.apply}\n    \n    def update_params(self, params, **kwargs):\n        if hasattr(self, \"interpolation\"):\n            params[\"interpolation\"] = self.interpolation\n        if hasattr(self, \"fill_value\"):\n            params[\"fill_value\"] = self.fill_value\n        return params\n\n    def get_sentences(self, text, lang='en'):\n        return sent_tokenize(text, self.LANGS.get(lang, 'english'))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## So let me implement some nlp \"albumentations\" :D"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ShuffleSentencesTransform(NLPTransform):\n    \"\"\" Do shuffle by sentence \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ShuffleSentencesTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        sentences = self.get_sentences(text, lang)\n        random.shuffle(sentences)\n        return ' '.join(sentences), lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = ShuffleSentencesTransform(p=1.0)\n\ntext = '<Sentence1>. <Sentence2>. <Sentence3>. <Sentence4>. <Sentence5>. <Sentence6>.'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ExcludeDuplicateSentencesTransform(NLPTransform):\n    \"\"\" Exclude equal sentences \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeDuplicateSentencesTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        sentences = []\n        for sentence in self.get_sentences(text, lang):\n            sentence = sentence.strip()\n            if sentence not in sentences:\n                sentences.append(sentence)\n        return ' '.join(sentences), lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = ExcludeDuplicateSentencesTransform(p=1.0)\n\ntext = '<Sentence1>. <Sentence2>. <Sentence4>. <Sentence4>. <Sentence5>. <Sentence5>.'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ExcludeNumbersTransform(NLPTransform):\n    \"\"\" exclude any numbers \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeNumbersTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'[0-9]', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = ExcludeNumbersTransform(p=1.0)\n\ntext = '<Word1> <Word2> <Word3> <Word4> <Word5> <Word6> <Word7> <Word8> <Word9> <Word10>'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ExcludeHashtagsTransform(NLPTransform):\n    \"\"\" Exclude any hashtags with # \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeHashtagsTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'#[\\S]+\\b', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = ExcludeHashtagsTransform(p=1.0)\n\ntext = '<Word1> <Word2> <Word3> #kaggle <Word4> <Word5> <Word6> <Word7> <Word8> <Word9> <Word10>'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ExcludeUsersMentionedTransform(NLPTransform):\n    \"\"\" Exclude @users \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeUsersMentionedTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'@[\\S]+\\b', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = ExcludeUsersMentionedTransform(p=1.0)\n\ntext = '<Word1> <Word2> <Word3> @kaggle <Word4> <Word5> <Word6> <Word7> <Word8> <Word9> <Word10>'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ExcludeUrlsTransform(NLPTransform):\n    \"\"\" Exclude urls \"\"\"\n    def __init__(self, always_apply=False, p=0.5):\n        super(ExcludeUrlsTransform, self).__init__(always_apply, p)\n\n    def apply(self, data, **params):\n        text, lang = data\n        text = re.sub(r'https?\\S+', '', text)\n        text = re.sub(r'\\s+', ' ', text)\n        return text, lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = ExcludeUrlsTransform(p=1.0)\n\ntext = '<Word1> <Word2> <Word3> <Word4> https://www.kaggle.com/shonenkov/nlp-albumentations/ <Word6> <Word7> <Word8> <Word9> <Word10>'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class SwapWordsTransform(NLPTransform):\n    \"\"\" Swap words next to each other \"\"\"\n    def __init__(self, swap_distance=1, swap_probability=0.1, always_apply=False, p=0.5):\n        \"\"\"  \n        swap_distance - distance for swapping words\n        swap_probability - probability of swapping for one word\n        \"\"\"\n        super(SwapWordsTransform, self).__init__(always_apply, p)\n        self.swap_distance = swap_distance\n        self.swap_probability = swap_probability\n        self.swap_range_list = list(range(1, swap_distance+1))\n\n    def apply(self, data, **params):\n        text, lang = data\n        words = text.split()\n        words_count = len(words)\n        if words_count <= 1:\n            return text, lang\n\n        new_words = {}\n        for i in range(words_count):\n            if random.random() > self.swap_probability:\n                new_words[i] = words[i]\n                continue\n    \n            if i < self.swap_distance:\n                new_words[i] = words[i]\n                continue\n    \n            swap_idx = i - random.choice(self.swap_range_list)\n            new_words[i] = new_words[swap_idx]\n            new_words[swap_idx] = words[i]\n\n        return ' '.join([v for k, v in sorted(new_words.items(), key=lambda x: x[0])]), lang\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = SwapWordsTransform(p=1.0, swap_distance=1, swap_probability=0.2)\n\ntext = '<Word1> <Word2> <Word3> <Word4> <Word5> <Word6> <Word7> <Word8> <Word9> <Word10>'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CutOutWordsTransform(NLPTransform):\n    \"\"\" Remove random words \"\"\"\n    def __init__(self, cutout_probability=0.05, always_apply=False, p=0.5):\n        super(CutOutWordsTransform, self).__init__(always_apply, p)\n        self.cutout_probability = cutout_probability\n\n    def apply(self, data, **params):\n        text, lang = data\n        words = text.split()\n        words_count = len(words)\n        if words_count <= 1:\n            return text, lang\n        \n        new_words = []\n        for i in range(words_count):\n            if random.random() < self.cutout_probability:\n                continue\n            new_words.append(words[i])\n\n        if len(new_words) == 0:\n            return words[random.randint(0, words_count-1)], lang\n\n        return ' '.join(new_words), lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = CutOutWordsTransform(p=1.0, cutout_probability=0.2)\n\ntext = '<Word1> <Word2> <Word3> <Word4> <Word5> <Word6> <Word7> <Word8> <Word9> <Word10>'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class AddNonToxicSentencesTransform(NLPTransform):\n    \"\"\" Add random non toxic statement \"\"\"\n    def __init__(self, non_toxic_sentences, sentence_range=(1, 3), always_apply=False, p=0.5):\n        super(AddNonToxicSentencesTransform, self).__init__(always_apply, p)\n        self.sentence_range = sentence_range\n        self.non_toxic_sentences = non_toxic_sentences\n\n    def apply(self, data, **params):\n        text, lang = data\n\n        sentences = self.get_sentences(text, lang)\n        for i in range(random.randint(*self.sentence_range)):\n            sentences.append(random.choice(self.non_toxic_sentences))\n        \n        random.shuffle(sentences)\n        return ' '.join(sentences), lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"usage example:"},{"metadata":{"trusted":true},"cell_type":"code","source":"nlp_transform = NLPTransform()\n\ndf = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv', nrows=1000)\ndf = df[df.toxic == 0]\ndf['lang'] = 'en'\nnon_toxic_sentences = set()\nfor comment_text in tqdm(df['comment_text'], total=df.shape[0]):\n    non_toxic_sentences.update(nlp_transform.get_sentences(comment_text), 'en')\n\ntransform = AddNonToxicSentencesTransform(non_toxic_sentences=list(non_toxic_sentences), p=1.0, sentence_range=(1,2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"text = '<Sentence1>. <Sentence2>. <Sentence4>. <Sentence4>. <Sentence5>. <Sentence5>.'\nlang = 'en'\n\ntransform(data=(text, lang))['data'][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Lets I show example for usage these classes for retrieving data using PyTorch Dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import albumentations\n\ndef get_train_transforms():\n    return albumentations.Compose([\n        ExcludeDuplicateSentencesTransform(p=0.9),  # here not p=1.0 because your nets should get some difficulties\n        albumentations.OneOf([\n            AddNonToxicSentencesTransform(non_toxic_sentences=list(non_toxic_sentences), p=0.8, sentence_range=(1,3)),\n            ShuffleSentencesTransform(p=0.8),\n        ]),\n        ExcludeNumbersTransform(p=0.8),\n        ExcludeHashtagsTransform(p=0.5),\n        ExcludeUsersMentionedTransform(p=0.9),\n        ExcludeUrlsTransform(p=0.9),\n        CutOutWordsTransform(p=0.1),\n        SwapWordsTransform(p=0.1),\n    ])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.utils.data import Dataset\n\nclass DatasetRetriever(Dataset):\n\n    def __init__(self, df, train_transforms=None):\n        self.comment_texts = df['comment_text'].values\n        self.langs = df['lang'].values\n        self.train_transforms = train_transforms\n\n    def __len__(self):\n        return self.comment_texts.shape[0]\n\n    def __getitem__(self, idx):\n        text = self.comment_texts[idx]\n        lang = self.langs[idx]\n        if self.train_transforms:\n            text, _ = self.train_transforms(data=(text, lang))['data']\n        return text","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"dataset = DatasetRetriever(df, train_transforms=get_train_transforms())\nfor albumentation_text in tqdm(dataset, total=len(dataset)):\n    pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Thank you for reading my kernel!\n\nI have shown great tools for you. \nAnd.. If you like this format of notebooks I would like continue to make kernels with realizations of my ideas.\n\n\nP.S. Method \"get_train_transforms\" is used only as example for you, my friends. You should get own collection augmentations :) "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}