{"cells":[{"metadata":{},"cell_type":"markdown","source":"## About this notebook\n\n*[Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)* is the 3rd annual competition organized by the Jigsaw team. It follows *[Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)*, the original 2018 competition, and *[Jigsaw Unintended Bias in Toxicity Classification](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification)*, which required the competitors to consider biased ML predictions in their new models. This year, the goal is to use english only training data to run toxicity predictions on many different languages, which can be done using multilingual models, and speed up using TPUs.\n\nMany awesome notebooks has already been made so far. Many of them used really cool technologies like [Pytorch XLA](https://www.kaggle.com/theoviel/bert-pytorch-huggingface-starter). This notebook instead aims at constructing a **fast, concise, reusable, and beginner-friendly model scaffold**. \n\n**THIS DOES NOT USE ANY TRANSLATED DATA, BUT IT DOES TRAIN ON THE VALIDATION SET.**\n\n\n### References\n* Original Author: [@xhlulu](https://www.kaggle.com/xhlulu/)\n* Original notebook: [Link](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input/jigsawtanslatedgoogle'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\n\nfrom tensorflow.keras.layers import Dense, Input,concatenate,Bidirectional, LSTM,MaxPool1D,MaxPool3D,GlobalMaxPooling1D,GlobalAveragePooling1D\nfrom tensorflow.keras.optimizers import Adam,SGD\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\n#from kaggle_datasets import KaggleDatasets\nimport transformers\nfrom transformers import *\nfrom transformers import TFAutoModel, AutoTokenizer,AutoModel\nfrom tqdm.notebook import tqdm\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors\ndef seed_everything(seed=0):\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\n\nSEED = 42\nseed_everything(SEED)\n#from kaggle_datasets import KaggleDatasets\n\nfrom tqdm.notebook import tqdm\nfrom tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors\nfrom sklearn.metrics import f1_score\nimport re\n\n\n\n# from pandarallel import pandarallel\n\n# pandarallel.initialize(nb_workers=2, progress_bar=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n    \"\"\"\n    https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras\n    \"\"\"\n    tokenizer.enable_truncation(max_length=maxlen)\n    tokenizer.enable_padding(max_length=maxlen)\n    all_ids = []\n    \n    for i in tqdm(range(0, len(texts), chunk_size)):\n        text_chunk = texts[i:i+chunk_size].tolist()\n        encs = tokenizer.encode_batch(text_chunk)\n        all_ids.extend([enc.ids for enc in encs])\n    \n    return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def regular_encode(texts, tokenizer, maxlen=512):\n    enc_di = tokenizer.batch_encode_plus(\n        texts, \n        return_attention_masks=True, \n        return_token_type_ids=True,\n        pad_to_max_length=True,\n        max_length=maxlen\n     )\n    #,np.asarray(enc_di['attention_masks'],dtype=np.int32)\n    return np.array(enc_di['input_ids'],dtype=np.int32),np.array(enc_di['attention_mask'],dtype=np.int32),np.array(enc_di['token_type_ids'],dtype=np.int32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras import backend as K\n\ndef focal_loss(gamma=2., alpha=.2):\n    def focal_loss_fixed(y_true, y_pred):\n        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n        return -K.mean(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) - K.mean((1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n    return focal_loss_fixed","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer,max_len=512):\n    input_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")#input_word_ids\n    input_mask = Input(shape=(max_len,), dtype=tf.int32, name=\"input_mask\")\n    segment_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"segment_ids\")\n    sequence_output = transformer((input_ids,input_mask,segment_ids))[0]\n    #cls_token = sequence_output[:, 0, :]\n    gp = GlobalMaxPooling1D()(sequence_output)\n    ap = GlobalAveragePooling1D()(sequence_output)\n    stack = concatenate([gp,ap],axis=1)\n    out = Dense(1, activation='sigmoid')(stack)\n    model = Model(inputs=[input_ids,input_mask,segment_ids], outputs=out)\n    model.compile(Adam(lr=0.2e-5),loss='binary_crossentropy', metrics=['accuracy',tf.keras.metrics.AUC()]) \n    return model\n#'binary_crossentropy'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Configs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\n#GCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n# Configuration\nEPOCHS = 2\nBATCH_SIZE = 16* strategy.num_replicas_in_sync\nMAX_LEN = 224\nMODEL = 'jplu/tf-xlm-roberta-large'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create fast tokenizer","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = AutoTokenizer.from_pretrained(MODEL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load text data into memory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\ntrain2.toxic=(train2.toxic>=0.5).astype(float)\n#train3 = pd.read_csv(\"/kaggle/input/quora-train/quora_train.csv\")\ntrain3 = pd.read_csv(\"/kaggle/input/jigsawtanslatedgoogle/jigsaw-unintended-bias-train_tr_clean.csv\")\ntrain3.toxic=(train3.toxic>=0.5).astype(float)\ntrain4 = pd.read_csv(\"/kaggle/input/jigsawtanslatedgoogle/jigsaw-unintended-bias-train_it_clean.csv\")\ntrain4.toxic=(train4.toxic>=0.5).astype(float)\ntrain5 = pd.read_csv(\"/kaggle/input/jigsawtanslatedgoogle/jigsaw-unintended-bias-train_pt_clean.csv\")\ntrain5.toxic=(train5.toxic>=0.5).astype(float)\ntrain6 = pd.read_csv(\"/kaggle/input/jigsawtanslatedgoogle/jigsaw-unintended-bias-train_ru_clean.csv\")\ntrain6.toxic=(train6.toxic>=0.5).astype(float)\ntrain7 = pd.read_csv(\"/kaggle/input/jigsawtanslatedgoogle/jigsaw-unintended-bias-train_fr_clean.csv\")\ntrain7.toxic=(train7.toxic>=0.5).astype(float)\ntrain8 = pd.read_csv(\"/kaggle/input/jigsawtanslatedgoogle/jigsaw-unintended-bias-train_es_clean.csv\")\ntrain8.toxic=(train8.toxic>=0.5).astype(float)\nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Combine train1 with a subset of train2\n# train = pd.concat([\n#     #train1[['comment_text', 'toxic']],\n#     train2[['comment_text', 'toxic']].query('toxic==1').sample(n=10000, random_state=32423),\n#     train2[['comment_text', 'toxic']].query('toxic==0').sample(n=10000, random_state=35345),   \n# ])\ntrain_tr = pd.concat([\n    train3[['comment_text', 'toxic']].query('toxic==1').sample(n=40000, random_state=35534),\n    train3[['comment_text', 'toxic']].query('toxic==0').sample(n=140000, random_state=56867),   \n])\ntrain_it = pd.concat([\n    train4[['comment_text', 'toxic']].query('toxic==1').sample(n=40000, random_state=45653),\n    train4[['comment_text', 'toxic']].query('toxic==0').sample(n=140000, random_state=56756),   \n])\ntrain_pt = pd.concat([\n    train5[['comment_text', 'toxic']].query('toxic==1').sample(n=40000, random_state=43253),\n    train5[['comment_text', 'toxic']].query('toxic==0').sample(n=140000, random_state=45754),   \n])\ntrain_ru = pd.concat([\n    train6[['comment_text', 'toxic']].query('toxic==1').sample(n=40000, random_state=45654),\n    train6[['comment_text', 'toxic']].query('toxic==0').sample(n=140000, random_state=67677),   \n])\ntrain_fr = pd.concat([\n    train7[['comment_text', 'toxic']].query('toxic==1').sample(n=40000, random_state=46565),\n    train7[['comment_text', 'toxic']].query('toxic==0').sample(n=140000, random_state=567676),   \n])\ntrain_es = pd.concat([\n    train8[['comment_text', 'toxic']].query('toxic==1').sample(n=40000, random_state=345465),\n    train8[['comment_text', 'toxic']].query('toxic==0').sample(n=140000, random_state=567657),   \n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#清除不必要的内存\nimport gc \ndel train1,train2,train3,train4,train5,train6,train7,train8\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nplt.figure(figsize=(10, 5))\nplt.subplot(1, 2, 1)\nsns.countplot(train_tr['toxic'])\nplt.title('Target on training data')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n\n#x_train = regular_encode(train['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_train_tr = regular_encode(train_tr['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_train_it = regular_encode(train_it['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_train_pt = regular_encode(train_pt['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_train_ru = regular_encode(train_ru['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_train_fr = regular_encode(train_fr['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_train_es = regular_encode(train_es['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_valid = regular_encode(valid['comment_text'].values, tokenizer, maxlen=MAX_LEN)\nx_test = regular_encode(test['content'].values, tokenizer, maxlen=MAX_LEN)\n\n#y_train = train.toxic.values\ny_train_tr = train_tr.toxic.values\ny_train_it = train_it.toxic.values\ny_train_pt = train_pt.toxic.values\ny_train_ru = train_ru.toxic.values\ny_train_fr = train_fr.toxic.values\ny_train_es = train_es.toxic.values\ny_valid = valid.toxic.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build datasets objects","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_dataset = (\n#     tf.data.Dataset\n#     .from_tensor_slices((x_train, y_train))\n#     .shuffle(len(train))\n#     .batch(BATCH_SIZE)\n#     .repeat()\n#     .prefetch(AUTO)\n# )\ntrain_tr_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train_tr, y_train_tr))\n    .shuffle(len(train_tr))\n    .batch(BATCH_SIZE)\n    .repeat()\n    .prefetch(AUTO)\n)\ntrain_it_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train_it, y_train_it))\n    .shuffle(len(train_it))\n    .batch(BATCH_SIZE)\n    .repeat()\n    .prefetch(AUTO)\n)\ntrain_pt_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train_pt, y_train_pt))\n    .shuffle(len(train_pt))\n    .batch(BATCH_SIZE)\n    .repeat()\n    .prefetch(AUTO)\n)\ntrain_ru_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train_ru, y_train_ru))\n    .shuffle(len(train_ru))\n    .batch(BATCH_SIZE)\n    .repeat()\n    .prefetch(AUTO)\n)\ntrain_fr_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train_fr, y_train_fr))\n    .shuffle(len(train_fr))\n    .batch(BATCH_SIZE)\n    .repeat()\n    .prefetch(AUTO)\n)\ntrain_es_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train_es, y_train_es))\n    .shuffle(len(train_es))\n    .batch(BATCH_SIZE)\n    .repeat()\n    .prefetch(AUTO)\n)\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\n# test_dataset = (\n#     tf.data.Dataset\n#     .from_tensor_slices(x_test)\n#     .batch(BATCH_SIZE)\n# )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#清除不必要的内存\nimport gc \ndel x_train_tr,x_train_it,x_train_pt,x_train_ru,x_train_fr,x_train_es,y_train_tr,y_train_it,y_train_pt,y_train_ru,y_train_fr,y_train_es\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load model into the TPU","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = TFAutoModel.from_pretrained('/kaggle/input/jigsaw-mlm-finetuned-xlm-r-large/')\n    model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First, we train on the subset of the training set, which is completely in English.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#微调tr语言\nn_steps = train_tr.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_tr_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#微调it语言\nn_steps = train_it.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_it_dataset,\n    steps_per_epoch=n_steps, \n    validation_data=valid_dataset,\n    epochs=EPOCHS,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#微调it语言\nn_steps = train_pt.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_pt_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#微调ru语言\nn_steps = train_ru.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_ru_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#微调fr语言\nn_steps = train_fr.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_fr_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#微调es语言\nn_steps = train_es.shape[0] // BATCH_SIZE\ntrain_history = model.fit(\n    train_es_dataset,\n    steps_per_epoch=n_steps,\n    validation_data=valid_dataset,\n    epochs=EPOCHS,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n# n_steps = train.shape[0] // BATCH_SIZE\n# train_history = model.fit(\n#     train_dataset,\n#     steps_per_epoch=n_steps,\n#     validation_data=valid_dataset,\n#     epochs=EPOCHS,\n#     shuffle=False,\n# )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have pretty much saturated the learning potential of the model on english only data, we train it for one more epoch on the `validation` set, which is significantly smaller but contains a mixture of different languages.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nn_steps = valid.shape[0] // BATCH_SIZE\ntrain_history_2 = model.fit(\n    valid_dataset.repeat(),\n    steps_per_epoch=n_steps,\n    epochs=4,\n    shuffle=False,\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['toxic'] = model.predict(x_test, verbose=1)\nsub.to_csv('submission.csv', index=False)\nsub.head(20)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}