{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\nimport time\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Bert was the fundament for growing new better models, for crosslingual tasks as well. Researchers from Facebook proposed XLM-Roberta where XLM stands for crosslingual model, Roberta - Robustly Optimized training approach for Bert\n\nHere is the abstract:\nThis paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make XLM-R code, data, and models publicly available.\n\nThe main idea is that this model is trained on much larger amount of data, espicially for low-resouce languages, for longer time, higher learning rate and batch size,\nwhat makes it's power to generalize more. What else, training process also differs - you can find details in original papers, but what is really important for us : while training one sentence was appeared not in one language: for example it might look like concatenation of english sentence and its translation in spanish, which helps the model to understand how the same words and sentences but in different languages are connected.\n\nOriginal paper claims that this new trained model outperforms ordinal bert, that is why we will use it later in our research.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"But before we train the model we need to encode data in the specific format manually. This format is simillar to bert at most,\nbut as far as data source is much wider the backstanding vocabulary differs.\n\nEncoding algorithm is the following: first, texts are split into the tokens - token might look a word or its part. Then every token in sentence is converted to its unique index in backstanding vocabulary for future connection with its embedding in the model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This notebook will be quite short but still very important. Kaggle gives an opportunity to use TPU while training but it is limited by 30 hours per week. \nData clearing and encoding is a quite time-consuming step in running the notebook. That is why it'll be more efficient to make tokenization in the single notebook and then use its outputs for testing model in the different one.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Our goal is tokenization is to\n1. Firstly clear the texts out of not important data\n2. Encoded clear text with custom tokenizer and SEQUENCE_LENGTH","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = \"../input/jigsaw-multilingual-toxic-comment-classification\"\nsmall_path = \"jigsaw-toxic-comment-train.csv\"\nlarge_path = \"jigsaw-unintended-bias-train.csv\"\nval_path = \"validation.csv\"\ntest_path = \"test.csv\"\nSEQUENCE_LENGTH = 192","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also I added augmented toxic comments to the data as the output of my previous notebook\n\nhttps://www.kaggle.com/vgodie/class-balancing","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"small_ds = pd.read_csv(os.path.join(DATA_PATH, small_path), usecols=[\"comment_text\", \"toxic\"])\nlarge_ds = pd.read_csv(os.path.join(DATA_PATH, large_path), usecols=[\"comment_text\", \"toxic\"])\naug_ds = pd.read_csv(\"../input/class-balancing/aug.csv\")\nval_ds = pd.read_csv(os.path.join(DATA_PATH, val_path), usecols=[\"comment_text\", \"toxic\"])\ntest_ds = pd.read_csv(os.path.join(DATA_PATH, test_path))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_ds = aug_ds.sample(300000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Large train dataset has about 2M samples, however we have limited RAM on Kaggle, that is why we will subsample\nonly part of non-toxic examples","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"large_toxic = large_ds[large_ds[\"toxic\"] > 0.5].round()\nlarge_nontoxic = large_ds[large_ds[\"toxic\"] == 0].sample(600000)\n\nds = pd.concat((small_ds,\n              large_toxic,\n              large_nontoxic,\n              aug_ds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before encoding into indeces text must be cleaned. Firstly, the punctuation will be removed. Also numbers, emails,\nlinks and usernames in order to save only words","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£',\n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', '\\xa0', '\\t',\n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', '\\u3000', '\\u202f',\n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', '«',\n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def clean_text(text, lang='en'):\n    text = str(text)\n    text = re.sub(r'[0-9\"]', '', text)\n    text = re.sub(r'#[\\S]+\\b', '', text)\n    text = re.sub(r'@[\\S]+\\b', '', text)\n    text = re.sub(r'https?\\S+', '', text)\n    text = re.sub(r'\\s+', ' ', text)\n    text = re.sub(\"\\[\\[User.*\",'',text)\n    for punct in puncts:\n        text = text.replace(punct, \"\")\n    return text\n\ndef clean_data(df, text_label=\"comment_text\", train=True):\n    pos = 0\n    while pos < len(df):\n        temp = df[pos:pos + 10000].copy()\n        df[pos:pos+10000][text_label] = temp[text_label].apply(clean_text).values\n        pos += 10000\n        print(\"Processed\", pos, \"texts\" )\n    df[\"lens\"] = df[text_label].str.split().apply(len)\n    if train:\n        df = df[df[\"lens\"] > 0]\n    df.drop(\"lens\", axis=1, inplace=True)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cleaned_ds = clean_data(ds)\ncleaned_val = clean_data(val_ds)\ncleaned_test = clean_data(test_ds,text_label=\"content\", train=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(cleaned_ds) == len(test_ds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"from transformers import AutoTokenizer\n\nMODEL = \"xlm-roberta-large\"\ntokenizer = AutoTokenizer.from_pretrained(MODEL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then dowloading tokenizer with config for xlm-roberta-large split texts to tokens,\nthen convert them to indeces and pad those sentences that a shorter than max length","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def encode_comments(dataframe, tokenizer=tokenizer, max_len=SEQUENCE_LENGTH):\n        \n        pos = 0\n        start = time.time()\n        \n        while pos < len(dataframe):\n            temp = dataframe[pos:pos+10000].copy()\n            res = tokenizer.batch_encode_plus(temp.comment_text.values,\n                                              pad_to_max_length=True,\n                                              max_length = SEQUENCE_LENGTH,\n                                              return_attention_masks = False\n                                             )\n            if pos == 0:\n                ids = np.array(res[\"input_ids\"])\n                labels = temp.toxic.values\n            else:\n                ids = np.concatenate((ids, np.array(res[\"input_ids\"])))\n                labels = np.concatenate((labels, temp.toxic.values))\n            pos+=10000\n            print(\"Processed\", pos, \"elements\")\n        return ids, labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ids,labels = encode_comments(cleaned_ds)\nval_ids,val_labels = encode_comments(cleaned_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ids = tokenizer.batch_encode_plus(cleaned_test.content.values,\n                                      pad_to_max_length=True,\n                                      max_length=SEQUENCE_LENGTH,\n                                      return_attention_masks=False)[\"input_ids\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#save the results of tokenization for future use in the next notebooks\n\nnp.save(\"ids.npy\", ids)\nnp.save(\"labels.npy\", labels)\nnp.save(\"val_ids.npy\", val_ids)\nnp.save(\"val_labels.npy\", val_labels)\nnp.save(\"test_ids.npy\", test_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step is to build XLM-Roberta Model - wanna see results - see my next notebook\n\n","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}