{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Hack with Parallel Corpus\n\n\nHi everyone!\n\nMy name is Alex Shonenkov, recently I have created two kernels with my ideas about this competition: [[TPU-Inference] Super Fast XLMRoberta](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta) and [NLP Albumentations](https://www.kaggle.com/shonenkov/nlp-albumentations).\n\nI am very impressive after some discussions that we could find common ground!\nIt very charges for me, thank you all, my friends!\n\nToday I would like to share with you dataset and simple intresting approach for hacking language difficulties due to given competition data doesn't have samples for other languages.\n\nI want to tell you about [Parallel Corpus](https://en.wikipedia.org/wiki/Parallel_text) and about how we can get new toxic samples on other languages.\n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## MAIN IDEA\n\nIn the past in these competitions [Jigsaw Unintended Bias in Toxicity Classification](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification) and [Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) we (Kaggle Community) created good models for English language with ~99% AUC score.\n\nAlso I would like to say thanks a lot [OpenSubtitles Community](https://www.opensubtitles.org/) for opportunities using their [datasets](http://opus.nlpl.eu/OpenSubtitles-v2018.php).\nDatasets have others formats but I used .txt format. It is two files on two languages, each of them has text splitted by line.\n\n\nSo, main idea: lets load corpus `en <--> other_lang`, make prediction toxic for english language and match these toxic labels with some thresholds for samples in the second file that has another language! (Pseudo-Labeling)\n\n\nP.S. First of all I used google to know information about using \"Mashine Translation\" for creating their datasets and I found information about OpenSubtitles Community [fighting](https://forum.opensubtitles.org/viewtopic.php?f=1&t=1969#p6752) with loaded subtitles created by \"Mashine Translation\""},{"metadata":{},"cell_type":"markdown","source":"### Implementation of this technique\n\nThis Corpus is very huge, but I would like to make demo on small corpus :)"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install transformers > /dev/null\n!pip install pandarallel > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport os\nos.environ['XLA_USE_BF16'] = \"1\"\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.utils.data.sampler import SequentialSampler\n\nimport time\nimport random\nfrom datetime import datetime\nfrom tqdm import tqdm\ntqdm.pandas()\n\nfrom transformers import XLMRobertaModel, XLMRobertaTokenizer, XLMRobertaConfig\nfrom transformers import AdamW, get_linear_schedule_with_warmup, get_constant_schedule\n\nimport re\n\n# !pip install nltk > /dev/null\nimport nltk\nnltk.download('punkt')\n\nfrom nltk import sent_tokenize\n\nfrom pandarallel import pandarallel\n\npandarallel.initialize(nb_workers=2, progress_bar=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget -O en-ru.txt.zip http://opus.nlpl.eu/download.php?f=OpenSubtitles/v2018/moses/en-ru.txt.zip\n!unzip en-ru.txt.zip\n!rm en-ru.txt.zip\n!rm OpenSubtitles.en-ru.ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!head -10 OpenSubtitles.en-ru.en","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!head -10 OpenSubtitles.en-ru.ru","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, above you can see EN and RU first 10 lines. I am from Russia so I can say that it is correct translation for my language. You can try another language if you want. "},{"metadata":{"trusted":true},"cell_type":"code","source":"ru_file = open('./OpenSubtitles.en-ru.ru', 'r')\nen_file = open('./OpenSubtitles.en-ru.en', 'r')\n\ndataset = {'en': [],'ru': []}\n\ntotal_lines = 10000\nfor i in tqdm(range(total_lines), total=total_lines):\n    ru_text = ru_file.readline()\n    en_text = en_file.readline()\n\n    if not en_text and not ru_text:\n        # one of file is finished\n        break\n\n    ru_text = ru_text.strip()\n    en_text = en_text.strip()\n    if not en_text or not ru_text:\n        continue\n\n    dataset['ru'].append(ru_text)\n    dataset['en'].append(en_text)\n    \nru_file, en_file = None, None\n\ndel ru_file\ndel en_file\n\ndf = pd.DataFrame(dataset)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Language detection\n\nI observed not only chosen language in samples. So I recommend do checking lang. For example using [SpaCY](https://spacy.io/usage/models) with multilang model. But this is not necessary."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install spacy > /dev/null\n!pip install spacy_cld > /dev/null\n!python -m spacy download xx_ent_wiki_sm > /dev/null\n\nimport spacy\nfrom spacy_cld import LanguageDetector\nimport xx_ent_wiki_sm\n\nnlp = xx_ent_wiki_sm.load()\nlanguage_detector = LanguageDetector()\nnlp.add_pipe(language_detector)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lang_score(text):\n    try:\n        doc = nlp(str(text))\n        language_scores = doc._.language_scores\n        return language_scores.get('ru', 0)\n    except Exception:\n        return 0\n\ntext = df.iloc[0]['ru']\nprint(f'[{get_lang_score(text)}]', text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['lang_score'] = df['ru'].parallel_apply(get_lang_score)\ndf['lang_score'].hist(bins=100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df[df['lang_score'] > 0.8]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I recommend you use good model for strong single English language with good accuracy. But It is only demo, so I can relax :D Let me use model from [my kernel](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta), for example using GPU."},{"metadata":{"trusted":true},"cell_type":"code","source":"MAX_LENGTH = 224\nBACKBONE_PATH = '../input/multitpu-inference'\nCHECKPOINT_PATH = '../input/multitpu-inference/checkpoint-xlm-roberta.bin'\n\nLANGS = {\n    'en': 'english',\n    'it': 'italian', \n    'fr': 'french', \n    'es': 'spanish',\n    'tr': 'turkish', \n    'ru': 'russian',\n    'pt': 'portuguese'\n}\n\ndef get_sentences(text, lang='en'):\n    return sent_tokenize(text, LANGS.get(lang, 'english'))\n\ndef exclude_duplicate_sentences(text, lang='en'):\n    sentences = []\n    for sentence in get_sentences(text, lang):\n        sentence = sentence.strip()\n        if sentence not in sentences:\n            sentences.append(sentence)\n    return ' '.join(sentences)\n\ndef clean_text(text, lang='en'):\n    text = str(text)\n    text = re.sub(r'[0-9\"]', '', text)\n    text = re.sub(r'#[\\S]+\\b', '', text)\n    text = re.sub(r'@[\\S]+\\b', '', text)\n    text = re.sub(r'https?\\S+', '', text)\n    text = re.sub(r'\\s+', ' ', text)\n    text = exclude_duplicate_sentences(text, lang)\n    return text.strip()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class DatasetRetriever(Dataset):\n\n    def __init__(self, df):\n        self.comment_texts = df['en'].values\n        self.ids = df.index.values\n        self.tokenizer = XLMRobertaTokenizer.from_pretrained(BACKBONE_PATH)\n\n    def get_tokens(self, text):\n        encoded = self.tokenizer.encode_plus(\n            text, \n            add_special_tokens=True, \n            max_length=MAX_LENGTH, \n            pad_to_max_length=True\n        )\n        return encoded['input_ids'], encoded['attention_mask']\n\n    def __len__(self):\n        return self.ids.shape[0]\n\n    def __getitem__(self, idx):\n        text = self.comment_texts[idx]\n\n        tokens, attention_mask = self.get_tokens(text)\n        tokens, attention_mask = torch.tensor(tokens), torch.tensor(attention_mask)\n\n        return self.ids[idx], tokens, attention_mask","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ndf['en'] = df.parallel_apply(lambda x: clean_text(x['en'], 'en'), axis=1)\ndf = df.drop_duplicates(subset='en')\ndf = df.drop_duplicates(subset='ru')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ToxicSimpleNNModel(nn.Module):\n\n    def __init__(self, backbone):\n        super(ToxicSimpleNNModel, self).__init__()\n        self.backbone = backbone\n        self.dropout = nn.Dropout(0.3)\n        self.linear = nn.Linear(\n            in_features=self.backbone.pooler.dense.out_features*2,\n            out_features=2,\n        )\n\n    def forward(self, input_ids, attention_masks):\n        bs, seq_length = input_ids.shape\n        seq_x, _ = self.backbone(input_ids=input_ids, attention_mask=attention_masks)\n        apool = torch.mean(seq_x, 1)\n        mpool, _ = torch.max(seq_x, 1)\n        x = torch.cat((apool, mpool), 1)\n        x = self.dropout(x)\n        return self.linear(x)\n\n\nbackbone = XLMRobertaModel(XLMRobertaConfig.from_pretrained(BACKBONE_PATH))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class GPUPredictor:\n    \n    def __init__(self, model):\n        self.device = torch.device('cuda:0')\n        self.model = model.to(self.device)\n\n    def run_inference(self, loader):\n        self.model.eval()\n        result = {'id': [], 'toxic': []}\n        for step, (ids, inputs, attention_masks) in tqdm(enumerate(loader), total=len(loader)):\n            with torch.no_grad():\n                inputs = inputs.to(self.device) \n                attention_masks = attention_masks.to(self.device)\n                outputs = self.model(inputs, attention_masks)\n                toxics = nn.functional.softmax(outputs, dim=1).data.cpu().numpy()[:,1]\n\n            result['id'].extend(ids.numpy())\n            result['toxic'].extend(toxics)\n\n        return pd.DataFrame(result)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"net = ToxicSimpleNNModel(backbone=backbone)\ncheckpoint = torch.load(CHECKPOINT_PATH)\nnet.load_state_dict(checkpoint);\n\ncheckpoint = None\ndel checkpoint","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = DatasetRetriever(df)\nloader = torch.utils.data.DataLoader(\n    dataset,\n    batch_size=16,\n    sampler=SequentialSampler(dataset),\n    pin_memory=False,\n    drop_last=False,\n    num_workers=2\n)\n\npredictor = GPUPredictor(net)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = predictor.run_inference(loader)\npredictions = predictions.set_index('id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Attention!\nThresholds I recommend choose on your validation set for English model! It is very important.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"df.loc[predictions.index, 'toxic'] = predictions['toxic']\ndf = df[(df['toxic'] > 0.9) | (df['toxic'] < 0.2)]\ndf.loc[:, 'toxic'] = df['toxic'].round().astype(int)\ndf['toxic'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, lets see on toxic samples:"},{"metadata":{"trusted":true},"cell_type":"code","source":"for _, row in df[df['toxic'] == 1].head(10).iterrows():\n    print('-'*10)\n    print('[EN]')\n    print(row['en'])\n    print('[RU]')\n    print(row['ru'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Smoothing\n\n\nSo, you can create your own pseudo-labeled multilingual toxic dataset. But how we can guarantee about correctness of marking!?\n\nI'm in a hurry to upset you: no way.\n\nSo I strongly recommend use label smoothing if you train on pseudo-labeled data.\n\nIf you don't know about label smoothing you should [read this](https://www.flixstock.com/label-smoothing-an-ingredient-of-higher-model-accuracy/)\n\n\n![](https://miro.medium.com/proxy/1*BQosKd3FHZWsfKRF6pdVtQ.png)\n"},{"metadata":{},"cell_type":"markdown","source":"## Thank you for reading my kernel!\n\nLet's go make high scoring solutions!\n\n\nP.S. Next my kernel will be about my training pipeline :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}