{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**In this kernel i will try to share my understanding and findings of cross lingual models.Feel free to correct me if I made any mistakes in this kernel.**\n\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**ChangeLog**\n* version 1 : training xlm roberta base for 1 epoch including validation english data in training set\n* version 2 :  training for 4 epoch (poor lb : close to 0.89)\n* version 3 : gamma=2.0 for focal loss, reducing train non toxic comment samples,removing translated validation data from train set,patience = 1 and epoch = 6 (bug)\n* version 4 : same as version 3 but trying to fix the  problem for 3 epoch(i was not monitoring val auc correctly in version 3) \n* version 5 : error\n* version 6 : patience = 2, extra 2 epochs using validation set\n* version 7 : using translated validation data + focal_loss(gamma=1.5,alpha = .25)\n* version 8 : More text cleaning(everything else is left same as version 7 so that we can compare version 7 with version 8's result for additional text cleaning) --> clean_text,replace_typical_misspell,handle_contractions,fix_quote\n* version 9 : as you can see now that in version 7 lb = 0.8987 and in version 8 we got lb 0.9151 where the only difference in version 9 was to add extra text cleaning techniques :) now in version 9 i will move to xlmr-large model  with maxlen = 192, BATCH_SIZE = 16 * strategy.num_replicas_in_sync and keeping everything else as it was before\n* version 10 : as we see version 9 model diverged and it seems like  loss=focal_loss(gamma=1.5,alpha = .25) worked just fine for xlmr base but not with large so will just try binary_crossentropy instead just to make sure whether or not my assumption is correct (leaving everything else as it is) saving model based on maximum validation accuracyand using EarlyStopping, ModelCheckpoint, LearningRateScheduler,training for 4 epochs \n* version 11 : trying to solve error of version 10\n* version 12 : trying to solve bugs of version 11 \n* version 13 : adding translated spanish data in training set \n* version 14 : oversampling validation english data, reducing spanish non english sample a bit for training for 5 epochs within 3 hours tpu limit (failed : couldn't finish commit within 3 hours  tpu limit)\n* version 15 : trying for 4 epoch and oversampling positive samples of translated validation set\n* version 16 : more data for training(958870 total) and using 2 epochs because it will take time [Note --> one thing i found in this competition is : large subset of trainset helps the learning algorithm perform better here instead of choosing small subset for long training] \n* version 17 : in version 16 you can see i got NotImplementedError after first epoch and i am unable to save best checkpoint,in previous versions i also tried  monitor='val_acc' and monitor='val_accuracy' but none of them saving best checkpoint for me,where am i making mistakes? in version 17 i will try to get rid of the error and will train again(if you know why i am unable to save best checkpoint please help me in the comment box) thanks in advance","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Imports**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n\nimport gc\nimport os\nimport time\nimport math\nimport random\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom datetime import date\nfrom transformers import *\nfrom sklearn.metrics import *\nfrom tqdm.notebook import tqdm\n\nimport torch\nimport torch.nn as nn\nimport torch.utils.data\nimport torch.nn.functional as F\n\n\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os\nimport gc\nimport re\nimport folium\n#import textstat\nfrom scipy import stats\nfrom colorama import Fore, Back, Style, init\n\nimport math\nimport numpy as np\nimport scipy as sp\nimport pandas as pd\n\nimport random\nimport networkx as nx\nfrom pandas import Timestamp\n\nfrom PIL import Image\nfrom IPython.display import SVG\nfrom keras.utils import model_to_dot\n\nimport requests\nfrom IPython.display import HTML\n\nimport seaborn as sns\nfrom tqdm import tqdm\nimport matplotlib.cm as cm\nimport matplotlib.pyplot as plt\n\ntqdm.pandas()\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport transformers\nimport tensorflow as tf\n\nfrom tensorflow.keras.callbacks import Callback\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, CSVLogger\n\nfrom tensorflow.keras.models import Model\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow.keras.optimizers import Adam\nfrom tokenizers import BertWordPieceTokenizer\nfrom tensorflow.keras.layers import Dense, Input, Dropout, Embedding\nfrom tensorflow.keras.layers import LSTM, GRU, Conv1D, SpatialDropout1D\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras import activations\nfrom tensorflow.keras import constraints\nfrom tensorflow.keras import initializers\nfrom tensorflow.keras import regularizers\n\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.optimizers import *\nfrom tensorflow.keras.activations import *\nfrom tensorflow.keras.constraints import *\nfrom tensorflow.keras.initializers import *\nfrom tensorflow.keras.regularizers import *\n\nfrom sklearn import metrics\nfrom sklearn.utils import shuffle\nfrom gensim.models import Word2Vec\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.feature_extraction.text import TfidfVectorizer,\\\n                                            CountVectorizer,\\\n                                            HashingVectorizer\n\nfrom nltk.stem.wordnet import WordNetLemmatizer \nfrom nltk.tokenize import word_tokenize\nfrom nltk.tokenize import TweetTokenizer  \n\nimport nltk\nfrom textblob import TextBlob\n\nfrom nltk.corpus import wordnet\nfrom nltk.corpus import stopwords\n#from googletrans import Translator\nfrom nltk import WordNetLemmatizer\n#from polyglot.detect import Detector\nfrom nltk.stem import WordNetLemmatizer\nfrom wordcloud import WordCloud, STOPWORDS\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\n\nstopword=set(STOPWORDS)\n\nlem = WordNetLemmatizer()\ntokenizer=TweetTokenizer()\n\nnp.random.seed(0)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Setup TPU configuration","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(tpu)\ntf.tpu.experimental.initialize_tpu_system(tpu)\nstrategy = tf.distribute.experimental.TPUStrategy(tpu)\nprint(strategy.num_replicas_in_sync)\n#GCS_DS_PATH = KaggleDatasets().get_gcs_path('jigsaw-multilingual-toxic-comment-classification')\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Part 1 : Understanding cross lingual models","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"the paper titled [**Cross-lingual Language Model Pretraining**](https://arxiv.org/abs/1901.07291) by Facebook AI, named XLM, presents an improved version of BERT to achieve state-of-the-art results in both classification and translation tasks.XLM uses a known pre-processing technique (BPE) and a dual-language training mechanism with BERT in order to learn relations between words in different languages. The model outperforms other models in a cross-lingual classification task (sentence entailment in 15 languages) and significantly improves machine translation when a pre-trained model is used for initialization of the translation model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# XLM is based on several key concepts:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Transformers** : The Transformer architecture is at the core of almost all the recent major developments in NLP.It introduced an attention mechanism that processes the entire text input simultaneously to learn contextual relations between words (or sub-words). A Transformer includes two parts — an encoder that reads the text input and generates a lateral representation of it (e.g. a vector for each word), and a decoder that produces the translated text from that representation.\n\n**A High-Level Look**\n\nLet’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.\n\n![](https://jalammar.github.io/images/t/the_transformer_3.png)\n\nPopping open that Optimus Prime goodness, we see an encoding component, a decoding component, and connections between them.\n\n![](https://jalammar.github.io/images/t/The_transformer_encoders_decoders.png)\n\nThe encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.\n\n![](https://jalammar.github.io/images/t/The_transformer_encoder_decoder_stack.png)\n\nThe encoders are all identical in structure (yet they do not share weights). Each one is broken down into two sub-layers:\n\n![](https://jalammar.github.io/images/t/Transformer_encoder.png)\n\nThe encoder’s inputs first flow through a self-attention layer – a layer that helps the encoder look at other words in the input sentence as it encodes a specific word. We’ll look closer at self-attention later in the post.\n\nThe outputs of the self-attention layer are fed to a feed-forward neural network. The exact same feed-forward network is independently applied to each position.\n\nThe decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence (similar what attention does in seq2seq models).\n\n![](https://jalammar.github.io/images/t/Transformer_decoder.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This Transformer architecture outperformed both RNNs and CNNs (convolutional neural networks). The computational resources required to train models were reduced as well. A win-win for everyone in NLP. Check out the below comparison:\n![](https://cdn.analyticsvidhya.com/wp-content/uploads/2019/03/transformercomparison.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The below animation wonderfully illustrates how Transformer works on a machine translation task:\n\n![](https://cdn.analyticsvidhya.com/wp-content/uploads/2019/03/transform20fps.gif)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The vanilla Transformer has only limited context of each word, i.e. only the predecessors of each word, in 2018 updated BERT used the Transformer’s encoder to learn a language model by masking (dropping) some of the words and then trying to predict them, allowing it to uses the entire context, i.e. words to the left and right of a masked word.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**How XLM works?**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The paper [**Cross-lingual Language Model Pretraining**](https://arxiv.org/abs/1901.07291) presents two innovative ideas — a new training technique of BERT for multilingual classification tasks and the use of BERT as initialization of machine translation models.\n\nThese are the language the XLM model supports: en-es-fr-de-zh-ru-pt-it-ar-ja-id-tr-nl-pl-simple-fa-vi-sv-ko-he-ro-no-hi-uk-cs-fi-hu-th-da-ca-el-bg-sr-ms-bn-hr-sl-zh_yue-az-sk-eo-ta-sh-lt-et-ml-la-bs-sq-arz-af-ka-mr-eu-tl-ang-gl-nn-ur-kk-be-hy-te-lv-mk-zh_classical-als-is-wuu-my-sco-mn-ceb-ast-cy-kn-br-an-gu-bar-uz-lb-ne-si-war-jv-ga-zh_min_nan-oc-ku-sw-nds-ckb-ia-yi-fy-scn-gan-tt-am.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The process of cross-lingual sentiment classification. We assume that the opinion units have already been determined. The English train set is used to train a classifier. The Spanish test set is mapped accordingly and the classifier is tested on this cross-lingual test set.check the below pictures : \n![](https://www.researchgate.net/profile/Jeremy_Barnes5/publication/309312650/figure/fig1/AS:669424235323406@1536614583578/The-process-of-cross-lingual-sentiment-classification-We-assume-that-the-opinion-units_W640.jpg)\nhere L1 means language 1 and L2 means language 2\n![](https://slideplayer.com/slide/12311059/73/images/13/Cross-lingual+Document+Classification.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* First, instead of using word or characters as the input of the model, it uses Byte-Pair Encoding (BPE) that splits the input into the most common sub-words across all languages, thereby increasing the shared vocabulary between languages.\n\n* Second, it upgrades the BERT architecture in two manners:\n\n     1.  Each training sample consists of the same text in two languages, whereas in BERT each sample is built from a single language. As in BERT, the goal of the model is to predict the masked tokens, however, with the new architecture, the model can use the context from one language to predict tokens in the other, as different words are masked words in each language (they are chosen randomly).   \n     \n     2. The model also receives the language ID and the order of the tokens in each language, i.e. the Positional Encoding, separately. The new metadata helps the model learn the relationship between related tokens in different languages.\n     ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**The upgraded BERT is denoted as Translation Language Modeling (TLM) while the “vanilla” BERT with BPE inputs is denoted as Masked Language Modeling (MLM).**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Training a Masked Language Model(MLM) for BERT","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"let's say the problem statement is : **Given an input sequence, we will randomly mask some words. The model then should predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**For quick demonstration purpose i will use code from analyticsvidhya's article [Introduction to PyTorch-Transformers: An Incredible Library for State-of-the-Art NLP (with Python code)](https://www.analyticsvidhya.com/blog/2019/07/pytorch-transformers-nlp-python/)**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First, let’s prepare a tokenized input from a text string using BertTokenizer:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nfrom pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM\n\nMODEL = 'jplu/tf-xlm-roberta-base'\n# Load pre-trained model tokenizer (vocabulary)\ntokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n\n# Tokenize input\ntext = \"[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]\"\ntokenized_text = tokenizer.tokenize(text)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is how our text looks like after tokenization:\n\n![](https://cdn.analyticsvidhya.com/wp-content/uploads/2019/07/Screenshot-from-2019-07-18-15-18-42.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The next step would be to convert this into a sequence of integers and create PyTorch tensors of them so that we can use them directly for computation:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Mask a token that we will try to predict back with `BertForMaskedLM`\nmasked_index = 8\ntokenized_text[masked_index] = '[MASK]'\nassert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']\n\n# Convert token to vocabulary indices\nindexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)\n# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)\nsegments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n\n# Convert inputs to PyTorch tensors\ntokens_tensor = torch.tensor([indexed_tokens])\nsegments_tensors = torch.tensor([segments_ids])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Notice that we have set [MASK] at the 8th index in the sentence which is the word ‘Hensen’. This is what our model will try to predict.\n\nNow that our data is rightly pre-processed for BERT, we will create a Masked Language Model. Let’s now use BertForMaskedLM to predict a masked token:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load pre-trained model (weights)\nmodel = BertForMaskedLM.from_pretrained('bert-base-uncased')\nmodel.eval()\ndevice  = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint(device)\n# If you have a GPU, put everything on cuda\ntokens_tensor = tokens_tensor.to(device)\nsegments_tensors = segments_tensors.to(device)\nmodel.to(device)\n\n# Predict all tokens\nwith torch.no_grad():\n    outputs = model(tokens_tensor, token_type_ids=segments_tensors)\n    predictions = outputs[0]\n\n# confirm we were able to predict 'henson'\npredicted_index = torch.argmax(predictions[0, masked_index]).item()\npredicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]\nassert predicted_token == 'henson'\nprint('Predicted token is:',predicted_token)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This was a small demo of training a Masked Language Model on a single input sequence.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The complete XLM model was trained by training both MLM and TLM and alternating between them.\n\n![](https://miro.medium.com/max/1400/0*lBYVNRe1esIXn1qE.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"To assess the contribution of the model, the paper presents its results on sentence entailment task (classify relationship between sentences) using XNLI dataset that includes sentences in 15 languages. The model significantly outperforms other prominent models","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Unsupervised Cross-lingual Representation Learning at Scale","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Abstract : \nThis paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make XLM-R code, data, and models publicly available.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Part 2 : Implementation using TPU Multiprocessing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Even though i am a pytorch lover but not sure if the video below is true for 2020 also or not....!!!!","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.core.display import HTML\n\n\nHTML('''<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/744f60NyAgc\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>''')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**implementation  is adapted from @tarunpaparaju's kernel  [Jigsaw Multilingual Toxicity : EDA + Models](https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models)**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"XLM-R handles the following 100 languages: Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Basque, Belarusian, Bengali, Bengali Romanized, Bosnian, Breton, Bulgarian, Burmese, Burmese, Catalan, Chinese (Simplified), Chinese (Traditional), Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Hausa, Hebrew, Hindi, Hindi Romanized, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish (Kurmanji), Kyrgyz, Lao, Latin, Latvian, Lithuanian, Macedonian, Malagasy, Malay, Malayalam, Marathi, Mongolian, Nepali, Norwegian, Oriya, Oromo, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Sanskri, Scottish, Gaelic, Serbian, Sindhi, Sinhala, Slovak, Slovenian, Somali, Spanish, Sundanese, Swahili, Swedish, Tamil, Tamil Romanized, Telugu, Telugu Romanized, Thai, Turkish, Ukrainian, Urdu, Urdu Romanized, Uyghur, Uzbek, Vietnamese, Welsh, Western, Frisian, Xhosa, Yiddish.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\ntrain2.toxic = train2.toxic.round().astype(int)\n\ntrain3 = pd.read_csv('../input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-es-cleaned.csv')\n\ntrain4 = pd.read_csv('../input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-es-cleaned.csv')\n\nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')\n\ntoxic = len(train2[['comment_text', 'toxic']].query('toxic==1'))\n# Combine train1 with a subset of train2\ntrain = pd.concat([\n    train1[['comment_text', 'toxic']],\n    train2[['comment_text', 'toxic']].query('toxic==1'),\n    \n    train3[['comment_text', 'toxic']].query('toxic==0'),\n    train3[['comment_text', 'toxic']].query('toxic==1'),\n    \n    train4[['comment_text', 'toxic']].query('toxic==0'),\n    train4[['comment_text', 'toxic']].query('toxic==1'),\n    \n    \n    train2[['comment_text', 'toxic']].query('toxic==0').sample(n=(toxic+(toxic//3)), random_state=101)\n])\n\ntest_data = test\ntrain_data = train\n\nmaxlen = 96","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nplt.figure(figsize=(10, 5))\nplt.subplot(1, 2, 1)\nsns.countplot(train_data['toxic'])\nplt.title('Target on training data')\n\nplt.subplot(1, 2, 2)\nsns.countplot(valid['toxic'])\nplt.title('Target on validation data')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"df_valid_en = pd.read_csv('../input/val-en-df/validation_en.csv')\n\n\n\ndf_valid_en = df_valid_en.drop(columns=['id', 'comment_text','lang'])\ndf_valid_en = df_valid_en.rename(columns={\"comment_text_en\": \"comment_text\"})\ncolumns_titles = [\"comment_text\",\"toxic\"]\ndf_valid_en=df_valid_en.reindex(columns=columns_titles)\n\n'''df2 = df_valid_en[df_valid_en.toxic == 1]\ndf_valid_en = pd.concat([df_valid_en,df2])\ndf_valid_en = pd.concat([df_valid_en,df2])  #doubling twice\n'''\n\nprint(df_valid_en.head(3))\n\ntrain_data = pd.concat([train_data , df_valid_en], axis=0).reset_index(drop=True)\ntrain_data = train_data.sample(frac=1).reset_index(drop=True)\n\nprint(len(train2[['comment_text', 'toxic']].query('toxic==1')))\nprint(len(train_data))\ntrain_data.head()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Clean the text (remove usernames and links)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val = valid\ntrain = train_data\n\ndef clean(text):\n    text = text.fillna(\"fillna\").str.lower()\n    text = text.map(lambda x: re.sub('\\\\n',' ',str(x)))\n    text = text.map(lambda x: re.sub(\"\\[\\[User.*\",'',str(x)))\n    text = text.map(lambda x: re.sub(\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\",'',str(x)))\n    text = text.map(lambda x: re.sub(\"\\(http://.*?\\s\\(http://.*\\)\",'',str(x)))\n    return text\n\nval[\"comment_text\"] = clean(val[\"comment_text\"])\ntest_data[\"content\"] = clean(test_data[\"content\"])\ntrain[\"comment_text\"] = clean(train[\"comment_text\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# More Text Cleaning\n\n**applying text cleaning techniques like clean_text,replace_typical_misspell,handle_contractions,fix_quote \non train,test and validation set**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/chenshengabc/from-quest-encoding-ensemble-a-little-bit-differen\n\npuncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£',\n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', '\\xa0', '\\t',\n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', '\\u3000', '\\u202f',\n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', '«',\n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\n\nmispell_dict = {\"aren't\" : \"are not\",\n\"can't\" : \"cannot\",\n\"couldn't\" : \"could not\",\n\"couldnt\" : \"could not\",\n\"didn't\" : \"did not\",\n\"doesn't\" : \"does not\",\n\"doesnt\" : \"does not\",\n\"don't\" : \"do not\",\n\"hadn't\" : \"had not\",\n\"hasn't\" : \"has not\",\n\"haven't\" : \"have not\",\n\"havent\" : \"have not\",\n\"he'd\" : \"he would\",\n\"he'll\" : \"he will\",\n\"he's\" : \"he is\",\n\"i'd\" : \"I would\",\n\"i'd\" : \"I had\",\n\"i'll\" : \"I will\",\n\"i'm\" : \"I am\",\n\"isn't\" : \"is not\",\n\"it's\" : \"it is\",\n\"it'll\":\"it will\",\n\"i've\" : \"I have\",\n\"let's\" : \"let us\",\n\"mightn't\" : \"might not\",\n\"mustn't\" : \"must not\",\n\"shan't\" : \"shall not\",\n\"she'd\" : \"she would\",\n\"she'll\" : \"she will\",\n\"she's\" : \"she is\",\n\"shouldn't\" : \"should not\",\n\"shouldnt\" : \"should not\",\n\"that's\" : \"that is\",\n\"thats\" : \"that is\",\n\"there's\" : \"there is\",\n\"theres\" : \"there is\",\n\"they'd\" : \"they would\",\n\"they'll\" : \"they will\",\n\"they're\" : \"they are\",\n\"theyre\":  \"they are\",\n\"they've\" : \"they have\",\n\"we'd\" : \"we would\",\n\"we're\" : \"we are\",\n\"weren't\" : \"were not\",\n\"we've\" : \"we have\",\n\"what'll\" : \"what will\",\n\"what're\" : \"what are\",\n\"what's\" : \"what is\",\n\"what've\" : \"what have\",\n\"where's\" : \"where is\",\n\"who'd\" : \"who would\",\n\"who'll\" : \"who will\",\n\"who're\" : \"who are\",\n\"who's\" : \"who is\",\n\"who've\" : \"who have\",\n\"won't\" : \"will not\",\n\"wouldn't\" : \"would not\",\n\"you'd\" : \"you would\",\n\"you'll\" : \"you will\",\n\"you're\" : \"you are\",\n\"you've\" : \"you have\",\n\"'re\": \" are\",\n\"wasn't\": \"was not\",\n\"we'll\":\" will\",\n\"didn't\": \"did not\",\n\"tryin'\":\"trying\"}\n\n\ndef clean_text(x):\n    x = str(x).replace(\"\\n\",\"\")\n    for punct in puncts:\n        x = x.replace(punct, f' {punct} ')\n    return x\n\n\ndef clean_numbers(x):\n    x = re.sub('[0-9]{5,}', '#####', x)\n    x = re.sub('[0-9]{4}', '####', x)\n    x = re.sub('[0-9]{3}', '###', x)\n    x = re.sub('[0-9]{2}', '##', x)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from nltk.tokenize.treebank import TreebankWordTokenizer\ntokenizer = TreebankWordTokenizer()\n\ndef handle_contractions(x):\n    x = tokenizer.tokenize(x)\n    return x\n\ndef fix_quote(x):\n    x = [x_[1:] if x_.startswith(\"'\") else x_ for x_ in x]\n    x = ' '.join(x)\n    return x\n\ndef _get_mispell(mispell_dict):\n    mispell_re = re.compile('(%s)' % '|'.join(mispell_dict.keys()))\n    return mispell_dict, mispell_re\n\n\ndef replace_typical_misspell(text):\n    mispellings, mispellings_re = _get_mispell(mispell_dict)\n\n    def replace(match):\n        return mispellings[match.group(0)]\n\n    return mispellings_re.sub(replace, text)\n\n\ndef clean_data(df, columns: list):\n    for col in columns:\n#         df[col] = df[col].apply(lambda x: clean_numbers(x))\n        df[col] = df[col].apply(lambda x: clean_text(x.lower())) \n        df[col] = df[col].apply(lambda x: replace_typical_misspell(x))\n        df[col] = df[col].apply(lambda x: handle_contractions(x))  \n        df[col] = df[col].apply(lambda x: fix_quote(x))   \n    \n    return df\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ninput_columns = [\n    'comment_text'   \n]\n\n'''applying text cleaning techniques like clean_text,replace_typical_misspell,handle_contractions,fix_quote \non train,test and validation set'''\n\ntrain = clean_data(train, input_columns ) \nval = clean_data(val, input_columns )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ninput_columns = [\n    'content'   \n]\ntest_data = clean_data(test_data, input_columns )\n\ndel tokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**we can see from above 2 cells that text cleaning for train,validation and test set takes 8+ minutes that means we are losing some of our vital times for training  on tpu which is 3 hours(max). so it would be a good idea if we create another kernel and save above 2 cells newly updated train,val and test_data as kernels output then using those files we can quickly import our new train,test and validation data here,which will save time for training model on TPU **","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Roc-Auc Evaluation metric**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class RocAucEvaluation(Callback):\n    def __init__(self, validation_data=(), interval=1):\n        super(Callback, self).__init__()\n\n        self.interval = interval\n        self.X_val, self.y_val = validation_data\n\n    def on_epoch_end(self, epoch, logs={}):\n        if epoch % self.interval == 0:\n            y_pred = self.model.predict(self.X_val, verbose=0)\n            score = roc_auc_score(self.y_val, y_pred)\n            print(\"\\n ROC-AUC - epoch: {:d} - score: {:.6f}\".format(epoch+1, score))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tokenize(encode) comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def regular_encode(texts, tokenizer, maxlen=512):\n    enc_di = tokenizer.batch_encode_plus(\n        texts, \n        return_attention_masks=False, \n        return_token_type_ids=False,\n        pad_to_max_length=True,\n        max_length=maxlen\n    )\n    \n    return np.array(enc_di['input_ids'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Load bert tokenizer**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"MODEL = 'jplu/tf-xlm-roberta-large'\n# First load the real tokenizer\ntokenizer = AutoTokenizer.from_pretrained(MODEL)\n\nsave_path = '/kaggle/working/xlmr_large/'\nif not os.path.exists(save_path):\n    os.makedirs(save_path)\ntokenizer.save_pretrained(save_path)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Encode comments**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nx_train = regular_encode(train.comment_text.astype(str), \n                      tokenizer, maxlen=maxlen)\nx_valid = regular_encode(val.comment_text.astype(str).values, \n                      tokenizer, maxlen=maxlen)\nx_test = regular_encode(test_data.content.astype(str).values, \n                     tokenizer, maxlen=maxlen)\n\ny_valid = val.toxic.values\ny_train = train.toxic.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**previously we lost 8+ minutes and here ~12 minutes,sum them and we have lost 20+ minutes, which is almost 1 epoch training time here..!! :(**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Train. validation and testing dataset**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Focal Loss","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras import backend as K\n\ndef focal_loss(gamma=2., alpha=.2):\n    def focal_loss_fixed(y_true, y_pred):\n        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n        return -K.mean(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) - K.mean((1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n    return focal_loss_fixed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the model and check summary","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, loss='binary_crossentropy', max_len=512):\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    x = tf.keras.layers.Dropout(0.2)(cls_token)\n    out = Dense(1, activation='sigmoid')(x)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=3e-5), loss=loss, metrics=[tf.keras.metrics.AUC()])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = transformers.TFXLMRobertaModel.from_pretrained(MODEL)\n    model = build_model(transformer_layer,loss='binary_crossentropy', max_len=maxlen)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Define Define ReduceLROnPlateau callback**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def callback():\n    cb = []\n\n    reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss',  \n                                    factor=0.3, patience=2, \n                                    verbose=1, mode='auto', \n                                    epsilon=0.0001, cooldown=1, min_lr=0.000001)\n    cb.append(reduceLROnPlat)\n    log = CSVLogger('log.csv')\n    cb.append(log)\n\n    RocAuc = RocAucEvaluation(validation_data=(x_valid, y_valid), interval=1)\n    cb.append(RocAuc)\n    \n    return cb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize model architecture","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model, dpi=70).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Learning rate schedule","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_lrfn(lr_start=0.000001, lr_max=0.000002, \n               lr_min=0.0000001, lr_rampup_epochs=7, \n               lr_sustain_epochs=0, lr_exp_decay=.87):\n    lr_max = lr_max * strategy.num_replicas_in_sync\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) * lr_exp_decay**(epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n        return lr\n    \n    return lrfn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(10, 7))\n\nlrfn = build_lrfn()\nplt.plot([i for i in range(35)], [lrfn(i) for i in range(35)]);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model_path = 'jigsawMultilingual.hdf5'\nmodel_path1 = '/kaggle/working/jigsawMultilingual.hdf5'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, LearningRateScheduler\n\ncheckpoint = ModelCheckpoint(model_path, monitor='val_accuracy', mode='max', save_best_only=True)\nes = EarlyStopping(monitor='val_accuracy', mode='max', patience=2, \n                   restore_best_weights=True, verbose=1)\nlr_callback = LearningRateScheduler(lrfn, verbose=1)\n\ncallback_list = [checkpoint,  lr_callback]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nN_STEPS = x_train.shape[0] // BATCH_SIZE\nEPOCHS = 2\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=N_STEPS,\n    validation_data=valid_dataset,\n    callbacks=callback_list,\n    epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if os.path.exists(model_path1):\n    model.load_weights(model_path1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we train it for 2 more epochs on the validation set, which is significantly smaller but contains a mixture of different languages.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nn_steps = x_valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    callbacks=callback_list,\n    epochs= EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save_weights('model.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if os.path.exists(model_path1):\n    model.load_weights(model_path1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"log_dir = \"/kaggle/working/log.csv\"\nif os.path.exists(log_dir):\n    os.remove(log_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Make Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('../input/jigsaw-multilingual-toxic-comment-classification/' + 'sample_submission.csv')\nsub['toxic'] = model.predict(test_dataset, verbose=1)\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**References**\n* [Jigsaw TPU: XLM-Roberta](https://www.kaggle.com/xhlulu/jigsaw-tpu-xlm-roberta)\n* [Jigsaw TPU: DistilBERT with Huggingface and Keras](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras)\n* [Jigsaw TPU: BERT with Huggingface and Keras](https://www.kaggle.com/miklgr500/jigsaw-tpu-bert-with-huggingface-and-keras)\n* [8 Excellent Pretrained Models to get you Started with Natural Language Processing (NLP)](https://www.analyticsvidhya.com/blog/2019/03/pretrained-models-get-started-nlp/)\n* [Introduction to PyTorch-Transformers: An Incredible Library for State-of-the-Art NLP (with Python code)](https://www.analyticsvidhya.com/blog/2019/07/pytorch-transformers-nlp-python/)\n\n* [facebookresearch/XLM](https://github.com/facebookresearch/XLM)\n\n* [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/)\n\n* [Exploring Distributional Representations and Machine Translation for Aspect-based Cross-lingual Sentiment Classification](https://www.researchgate.net/publication/309312650_Exploring_Distributional_Representations_and_Machine_Translation_for_Aspect-based_Cross-lingual_Sentiment_Classification)\n* [Fastai with 🤗 Transformers (BERT, RoBERTa, ...)](https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta)\n* [Jigsaw Multilingual Toxicity : EDA + Models](https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models)\n* [Flower Classification with TPUs - EDA and Baseline](https://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline/notebook)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}