{"cells":[{"metadata":{},"cell_type":"markdown","source":"**This notebook is used for predicting.**\n\nI'm grateful to Kaggle for such an amazing platform for learning and practicing. Jigsaw Multilingual Toxic Comment Classification is a great task for me to try my computational linguistic ideas.\n\nMy linguistic hypothesis:\n1. The toxic distributions may not be similar for different languages. Therefore, my English training corpus only involved 1:1 toxic vs non-toxic comments.\n2. Transferring any model trained on one language to another language may lead to some overfit or underfit. Therefore, my post processing included weights for different languages and fine tuning according to a list of profanity words.\n3. Different models may capture different cross-lingual aspects. Therefore, I tried to train many models of the same/different neural network structures and collect the best combination results.\n4. A toxic comment has at least one toxic word or sentence, while there exists no toxic elements in a non-toxic comment. In accordance with this hypothesis, I randomly generated more corpus for training.\n\nSummary of my approach:\n1. Employed Xlm-roberta-large for multilingual tokenizing.\n2. Defined two NN structures. Pure Text Model (PT): with pure texts as input, and Mixed Model (MX): with multi-inputs of pure texts and relevant language information about language types and whether the texts were translated from other languages.\n3. Employed pseudo-labelling for more training corpus. When my public score was below 0.9490, I chose texts with predicted labels either larger than 0.8 or smaller then 0.2. When my public score was above 0.9490, I used all pseudo labelled data.\n4. Generated more corpus for training. I used my best model to predict the toxicities for all sentences of each comment in the test data, chose the one with the highest predicted toxicity as a toxic sentence when the comment was labelled as toxic in my best submission; and chose the ones with predicted toxicities lower than 0.2 as non-toxic sentences when the comment was labelled as non-toxic in my best submission. I finally joined each toxic sentence with 5 different non-toxic sentences to generate a toxic corpus, and randomly joined some non-toxic sentences to generate a non-toxic corpus.\n5. For my final submission, I used 3 PT models and 2 MX models. 2 PT models were trained on English data, validation and pseudo labelled data, the other PT model and one MX model were trained on my generated multilingual data, and the other MX model was trained on English, translated multilingual, validation, pseudo labelled data. They were all used to predict on the original test data.\n6. I used Linear Regression to ensemble predictions from the 5 models, smoothed the results by multiplying language weights and  profanity weights. The Linear Regression model and smoothing weights were learned from 80% of my historical best predictions. At last, the results were blended with my historical best by weights of (0.2, 0.8).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**0. Set up the environment.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\nimport numpy as np\nfrom sklearn import metrics\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input, Concatenate\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nimport transformers\nfrom transformers import TFAutoModel, AutoTokenizer\nfrom tqdm.notebook import tqdm\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.linear_model import LinearRegression\n\n# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**1. Hyperparameters and useful functions.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n# Configuration\nEPOCHS = 2\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync#16\nMAX_LEN = 192\n\n# Load the transformer tokenizer\nMODEL = 'jplu/tf-xlm-roberta-large'\ntokenizer = AutoTokenizer.from_pretrained(MODEL)\n\n#Define encoder.\ndef regular_encode(texts, tokenizer, maxlen=512):\n    enc_di = tokenizer.batch_encode_plus(\n        texts, \n        return_attention_masks=False, \n        return_token_type_ids=False,\n        pad_to_max_length=True,\n        max_length=maxlen\n    )\n    \n    return np.array(enc_di['input_ids'])\n\n#Build a pure text model where language information is not considered.\ndef build_model_PT(transformer, max_len=512):\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=[tf.keras.metrics.AUC()])\n    \n    return model\n\n#Build a mixed model where language types are also featured.\ndef build_model_mix(transformer, max_len=512):\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    input_lang_tags = Input(shape=(4,), dtype=tf.float32, name=\"input_lang_tags\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    x = Concatenate()([cls_token, input_lang_tags])\n    out = Dense(1, activation='sigmoid')(x)\n    \n    model = Model(inputs=[input_word_ids, input_lang_tags], outputs=out)\n    \n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=[tf.keras.metrics.AUC()])\n    \n    return model\n\n#Function for coding language information.\ndef lang_embed(lang, tran):\n    lang_codes = {'en':'000', 'es':'100', 'fr':'010',\n                  'it':'001', 'pt':'110', 'ru':'101',\n                  'tr':'011'}\n    tran_codes = {'orig':'0', 'tran':'1'}\n    vec = lang_codes[lang]+tran_codes[tran]\n    vec = [int(v) for v in vec]\n    return vec\n\n#Function for cutting off the middle part of long texts.\ndef text_process(text):\n    ws = text.split(' ')\n    if(len(ws)>160):\n        text = ' '.join(ws[:160]) + ' ' + ' '.join(ws[-32:])\n    return text","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**2. Build the original and translated test data.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-test-translated/jigsaw_miltilingual_test_translated.csv')\n\ntest['content'] = test['content'].apply(lambda x: text_process(x))\n#test['translated'] = test['translated'].apply(lambda x: text_process(x))\n                       \nx_test = regular_encode(test.content.values, tokenizer, maxlen=MAX_LEN)\nlang_tag_test = np.array([lang_embed(row['lang'], 'orig') for _, row in test.iterrows()])\n\n#x_tran_test = regular_encode(test.translated.values, tokenizer, maxlen=MAX_LEN)\n#lang_tag_tran_test = np.array([lang_embed('en', 'tran') for _, row in test.iterrows()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**3. Load models and predict.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"***3.1 Predict with pure text models.***","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = TFAutoModel.from_pretrained(MODEL)\n    model = build_model_PT(transformer_layer, max_len=MAX_LEN)\n#model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Pure text model trained on generated multilingual data, pseudo labelled and validation data predicts on multilingual.\nmodel.load_weights('/kaggle/input/mymodels/mg2mp4.h5')\ntest['p1'] = model.predict(x_test, verbose=1)\n#test.to_csv('sub1.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Pure text model trained on English data, validation and pseudo labelled data predicts on multilingual.\nmodel.load_weights('/kaggle/input/mymodels/en2mp1.h5')\ntest['p2'] = model.predict(x_test, verbose=1)\n#test.to_csv('sub2.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Pure text model trained on English data, validation and pseudo labelled data predicts on multilingual.\nmodel.load_weights('/kaggle/input/mymodels/en2mp4.h5')\ntest['p3'] = model.predict(x_test, verbose=1)\n#test.to_csv('sub3.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***3.2 Predict with mixed language models.***","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Clear up the memory first.\ndel model\nfrom keras import backend as K\nimport gc\nK.clear_session()\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = TFAutoModel.from_pretrained(MODEL)\n    model = build_model_mix(transformer_layer, max_len=MAX_LEN)\n#model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Mixed model trained on generated multilingual test data, original validation and pseudo labelled data predicts on multilingual.\nmodel.load_weights('/kaggle/input/mymodels/mixmoriggen.h5')\ntest['p4'] = model.predict([x_test, lang_tag_test], verbose=1)\n#test.to_csv('sub4.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Mixed model trained on English, translated multilingual, validation, pseudo labelled and generated data, predicts on multilingual.\nmodel.load_weights('/kaggle/input/mymodels/mixmorigp3.h5')\ntest['p5'] = model.predict([x_test, lang_tag_test], verbose=1)\n#test.to_csv('sub5.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**4. Blend and smooth for submission.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"mybest = pd.read_csv('/kaggle/input/mybest/sub9523.csv')\n#Blend first.\nX = np.array([test.p1.values, test.p2.values, test.p3.values, test.p4.values, test.p5.values]).T\ny = mybest.toxic.values\nX_T, X_val, y_T, y_val = train_test_split(X, y, test_size=0.2)\n    \nmodel = LinearRegression()\nmodel.fit(X, y)\n    \nprds = model.predict(X_val)\nscore1 = roc_auc_score(y_val.round().astype(int), prds)\nscore2 = roc_auc_score(prds.round().astype(int), y_val)\nprint('Validation:', score1, score2)  \n\nprds = model.predict(X)\nscore1 = roc_auc_score(y.round().astype(int), prds)\nscore2 = roc_auc_score(prds.round().astype(int), y)\nprint('Again my best:', score1, score2)\ntest['prd'] = prds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Then smooth.\np = [1.3,0.6,0.8,0.5,0.6,0.6]\nout = []\nfor _, row in test.iterrows():\n    item = [row['id'], row['prd'], row['lang']]\n    if(item[2]=='es'):\n        if(item[1]<0.7):\n            item[1] *= p[0]\n    elif(item[2]=='fr'):\n        if(item[1]<0.7):\n            item[1] *= p[1]\n    elif(item[2]=='ru'):\n        if(item[1]<0.7):\n            item[1] *= p[2]\n    elif(item[2]=='it'):\n        if(item[1]<0.7):\n            item[1] *= p[3]\n    elif(item[2]=='tr'):\n        if(item[1]<0.7):\n            item[1] *= p[4]\n    elif(item[2]=='pt'):\n        if(item[1]<0.7):\n            item[1] *= p[5]\n\n    out.append([item[0], item[1]])\n\nof = pd.DataFrame(out, columns=['id', 'toxic'])\nprint(of.head())\nscore1 = roc_auc_score(mybest.toxic.round().astype(int), of.toxic.values)\nscore2 = roc_auc_score(of.toxic.round().astype(int), mybest.toxic.values)\nprint('%2.4f\\t%2.4f'%(100*score1, 100*score2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**5. Fine tune results with profanity list.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"dic = {}\noft = open('/kaggle/input/profanity/Profanity.txt', \"r\", encoding='utf8')\nfor l in oft:\n    ele = l.strip().lower().split(':')\n    dic[ele[0]] = ele[1]\noft.close()\n\nles, lit, ltr, lfr, lru, lpt = 1.2, 1.1, 1.3, 1.2, 1.2, 1.3\nof['content'] = test['content']\nof['tran'] = test['translated']\nof['lang'] = test['lang']\nout = []\nenpros = dic['en'].split(',')\n\nfor _, row in of.iterrows():\n    if(row['lang']=='es'):\n        lmd = les\n    elif(row['lang']=='it'):\n        lmd = lit\n    elif(row['lang']=='tr'):\n        lmd = ltr\n    elif(row['lang']=='fr'):\n        lmd = lfr\n    elif(row['lang']=='ru'):\n        lmd = lru\n    else:\n        lmd = lpt\n\n    item = [row['id'], row['toxic']]\n    if(item[1]<0.5):\n        for w in enpros:\n            if(str(row['tran']).lower().find(w)>=0):\n                item[1] *= 1.2\n                break\n\n        ws = dic[row['lang']].split(',')\n        for w in ws:\n            if(str(row['content']).lower().find(w)>=0):\n                item[1] *= lmd\n                break\n    out.append(item)\n\nof = pd.DataFrame(out, columns=['id', 'toxic'])\nscore1 = roc_auc_score(mybest.toxic.round().astype(int), of.toxic.values)\nscore2 = roc_auc_score(of.toxic.round().astype(int), mybest.toxic.values)\nprint('%2.4f\\t%2.4f'%(100*score1, 100*score2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**6. Ensemble with my historical best.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"of['toxic'] = mybest.toxic.values*0.8 + of.toxic.values*0.2\nscore1 = roc_auc_score(mybest.toxic.round().astype(int), of.toxic.values)\nscore2 = roc_auc_score(of.toxic.round().astype(int), mybest.toxic.values)\nprint('%2.4f\\t%2.4f'%(100*score1, 100*score2))\nprint(of.head())\nof.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}