{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\nThis notebook represents a template for the initial baseline that I used, for the languages that had a separate validation data ***(es, it, tr)***. <br>\nI was highly inspired by a few kernels (see references below), from which a big part of my baseline code is derived.\n\n[Notebook Jigsaw TPU: XLM-Roberta](https://www.kaggle.com/xhlulu/jigsaw-tpu-xlm-roberta) showcased a multilingual model as one of the first in the competition, and served as guideline for many of the competitors including myself. <br>\n\nThe main changes in my template baseline compared to reference:\n1. The translated dataset [Jigsaw Train Multilingual Coments (Google API)](https://www.kaggle.com/miklgr500/jigsaw-train-multilingual-coments-google-api) is used for training, instead of original English-only. In this example, I show how to train on the Spanish translated dataset <br>\n2. Making use of a custom training loop on TPU\n3. Parallel processing for faster tokenization\n4. Balancing the non-toxic vs. toxic comments as 4:1 (i.e 0.8 fraction of non-toxic and 0.2 fraction of toxic)\n5. Divide one epoch in several iterations\n6. Make test predictions using the model with best AUC on validation\n7. Printing out more metrics during training (TP/TN/FP/FN & AUC)\n8. Stage 2: further resume train on validation data with best model (higher score on LB) \n9. Further upgrade to initial baseline: predict with repeated psuedolabels\n\n\\* Note: This notebook is used for the case of the languages where a separate validation dataset is given (es, tr, it), where the validation dataset is used for validating the model. For the remaining languages (ru, pt, fr) a simple average was taken over several snapshots ensembles. This is shown in another notebook.\n\n\n### References: notebooks and guides\n1. [Jigsaw TPU: XLM-Roberta](https://www.kaggle.com/xhlulu/jigsaw-tpu-xlm-roberta)\n2. [Custom Training Loop with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/custom-training-loop-with-100-flowers-on-tpu)\n3. [Tensorflow guide (useful for balancing datasets)](https://www.tensorflow.org/guide/data)\n4. [Hugginface models](https://huggingface.co/models)\n\n### References: datasets\n1. [Jigsaw Train Multilingual Coments (Google API)](https://www.kaggle.com/miklgr500/jigsaw-train-multilingual-coments-google-api)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Imports","execution_count":null},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# General imports\nimport os\nimport numpy as np\nimport pandas as pd\nfrom tqdm.notebook import tqdm\nfrom joblib import Parallel, delayed\n\n# Tensorflow\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\n\n# Transformers\nfrom transformers import TFAutoModel, TFBertModel, AutoTokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU and configs","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Configuration parameters\nAUTO = tf.data.experimental.AUTOTUNE\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nMAX_LEN = 192\nMODEL = 'jplu/tf-xlm-roberta-large' # for BERT model replace by e.g. dccuchile/bert-base-spanish-wwm-uncased\nLANG = \"es\" # can be any of es, it, tr in this notebook\nCONSTANT_LR = 3e-6 # 3e-6 generally good. Set lower e.g. 1e-6 for more finetuning\nBALANCEMENT = [0.8, 0.2] # non-toxic vs. toxic\nBERT_MODEL = False # specify if the given model is a BERT model\nN_EPOCHS = 3 # 3-5 epochs are usually enough. Set higher e.g. 5 for more finetuning\nN_ITER_PER_EPOCH = 10\nPREDICT_START_ITER = 10 # start iteration to predict on test. best iterations found around +-20 (2 full epochs)\n\n# Upgrades\nSTAGE2 = True # resume training with checkpoint of best model\nREPEAT_PL = 0 # Upgrade: repeat PL with train (I repeated 6x on my last subs). Default=0 (no pseudolabels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions processing","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def regular_encode(texts, max_len):\n    \"\"\"\n    Tokenizing the texts into their respective IDs using regular batch encoding\n    \n    Accepts: * texts: the text to be tokenize\n             * max_len: max length of text\n    \n    Returns: * array of tokenized IDs \n    \"\"\"\n    enc_di = tokenizer.batch_encode_plus(\n        texts, \n        return_attention_masks=False, \n        return_token_type_ids=False,\n        pad_to_max_length=True,\n        max_length=max_len\n    )\n    \n    return np.array(enc_di['input_ids'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def parallel_encode(texts, max_len):\n    \"\"\"\n    Tokenizing the texts into their respective IDs using parallel processing\n    \n    Accepts: * texts: the text to be tokenized\n             * max_len: max length of text\n    \n    Returns: * array of tokenized IDs + the toxicity label  \n    \"\"\"\n    enc_di = tokenizer.encode_plus(\n        str(texts[0]),\n        return_attention_masks=False, \n        return_token_type_ids=False,\n        pad_to_max_length=True,\n        max_length=max_len\n    )\n    return np.array(enc_di['input_ids']), texts[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, max_len):\n    \"\"\"\n    Build the model by using transformer layer and simple CLS token\n    \n    Accepts: * transformer: transformer layer\n             * max_len: max length of text\n    \n    Returns: * model \n    \"\"\"\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    model = Model(inputs=input_word_ids, outputs=out)    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create fast tokenizer","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = AutoTokenizer.from_pretrained(MODEL)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load text data into memory","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(f\"/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-{LANG}-cleaned.csv\")\nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if REPEAT_PL:\n    sub = pd.read_csv(\"../input/multilingual-toxic-comments-training-data/test9500.csv\") # use one of earlier subs\n    sub[\"comment_text\"] = test[\"content\"]\n    sub = sub.loc[test[\"lang\"]==LANG].reset_index(drop=True)\n    sub_repeat = pd.concat([sub]*REPEAT_PL, ignore_index=True) # repeat PL multipe times for training\n    same_cols = [\"comment_text\", \"toxic\"]\n    train = pd.concat([train[same_cols], sub_repeat[same_cols]]).sample(frac=1).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get specific validation and test\nvalid = valid.loc[valid[\"lang\"]==LANG].reset_index(drop=True)\ntest = test.loc[test[\"lang\"]==LANG].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tokenize","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time \n# Tokenize train with parallel processing\nrows = zip(train['comment_text'].values.tolist(), train.toxic.values.tolist())\nx_y_train = Parallel(n_jobs=4, backend='multiprocessing')(delayed(parallel_encode)(row, max_len=MAX_LEN) for row in tqdm(rows))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = np.vstack(np.array(x_y_train)[:,0])\n\ny_train = np.array(x_y_train)[:,1]\ny_train = np.asarray(y_train).astype('float32').reshape((-1,1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Tokenize valid regular processing\nx_valid = regular_encode(valid.comment_text.values, max_len=MAX_LEN)\n\ny_valid = valid.toxic.values\ny_valid = np.asarray(y_valid).astype('float32').reshape((-1,1)) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nx_test = regular_encode(test.content.values, max_len=MAX_LEN)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build datasets objects","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train and valid dataset\ntrain_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .shuffle(buffer_size=len(x_train), seed = 18)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Balance the train dataset by creating seperate negative and positive datasets. \n# Note: tf.squeeze remove the added dim to labels\n# Example taken from https://www.tensorflow.org/guide/data\n\nnegative_ds = (\n  train_dataset\n    .filter(lambda _, y: tf.squeeze(y)==0)\n    .repeat())\n\npositive_ds = (\n  train_dataset\n    .filter(lambda _, y: tf.squeeze(y)==1)\n    .repeat())\n\nbalanced_ds = tf.data.experimental.sample_from_datasets(\n    [negative_ds, positive_ds], BALANCEMENT).batch(BATCH_SIZE) # Around 80%/20% to be expected for 0/1 labels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# distribute the datset according to the strategy\ntrain_dist_ds = strategy.experimental_distribute_dataset(balanced_ds)\nvalid_dist_ds = strategy.experimental_distribute_dataset(valid_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions TF custom training","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Instantiate metrics\nwith strategy.scope():\n    # Accuracy, AUC, loss train\n    train_accuracy = tf.keras.metrics.BinaryAccuracy()\n    train_auc = tf.keras.metrics.AUC()\n    train_loss = tf.keras.metrics.Sum()\n    \n    # Accuracy, AUC, loss valid\n    valid_accuracy = tf.keras.metrics.BinaryAccuracy()\n    valid_auc = tf.keras.metrics.AUC()\n    valid_loss = tf.keras.metrics.Sum()\n    \n    # TP, TN, FN, FP train\n    train_TP = tf.keras.metrics.TruePositives()\n    train_TN = tf.keras.metrics.TrueNegatives()\n    train_FP = tf.keras.metrics.FalsePositives()\n    train_FN = tf.keras.metrics.FalseNegatives()\n    \n    # TP, TN, FN, FP valid\n    valid_TP = tf.keras.metrics.TruePositives()\n    valid_TN = tf.keras.metrics.TrueNegatives()\n    valid_FP = tf.keras.metrics.FalsePositives()\n    valid_FN = tf.keras.metrics.FalseNegatives()\n    \n    # Loss function and optimizer\n    loss_fn = lambda a,b: tf.nn.compute_average_loss(tf.keras.losses.binary_crossentropy(a,b), global_batch_size=BATCH_SIZE)\n    optimizer = tf.keras.optimizers.Adam(learning_rate=CONSTANT_LR)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"@tf.function\ndef train_step(tokens, labels):\n    with tf.GradientTape() as tape:\n        probabilities = model(tokens, training=True)\n        loss = loss_fn(labels, probabilities)\n    grads = tape.gradient(loss, model.trainable_variables)\n    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n    \n    # update metrics\n    train_accuracy.update_state(labels, probabilities)\n    train_auc.update_state(labels, probabilities)\n    train_loss.update_state(loss)\n    \n    train_TP.update_state(labels, probabilities)\n    train_TN.update_state(labels, probabilities)\n    train_FP.update_state(labels, probabilities)\n    train_FN.update_state(labels, probabilities)\n    \n@tf.function\ndef valid_step(tokens, labels):\n    probabilities = model(tokens, training=False)\n    loss = loss_fn(labels, probabilities)\n    \n    # update metrics\n    valid_accuracy.update_state(labels, probabilities)\n    valid_auc.update_state(labels, probabilities)\n    valid_loss.update_state(loss)\n    \n    valid_TP.update_state(labels, probabilities)\n    valid_TN.update_state(labels, probabilities)\n    valid_FP.update_state(labels, probabilities)\n    valid_FN.update_state(labels, probabilities)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    if BERT_MODEL:\n        transformer_layer = TFBertModel.from_pretrained(MODEL, from_pt=True)\n    else:\n        transformer_layer = TFAutoModel.from_pretrained(MODEL)\n    model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Custom training loop","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"VALIDATION_STEPS = x_valid.shape[0] // BATCH_SIZE\nSTEPS_PER_EPOCH = x_train.shape[0] // (BATCH_SIZE*N_ITER_PER_EPOCH)\nprint(\"Steps per epoch:\", STEPS_PER_EPOCH)\nEPOCHS = N_EPOCHS*N_ITER_PER_EPOCH\n\nbest_auc = 0\nepoch = 0\n\npreds_all = []\nfor step, (tokens, labels) in enumerate(train_dist_ds):\n    # run training step\n    strategy.experimental_run_v2(train_step, args=(tokens, labels))\n    print('=', end='', flush=True)\n    \n    # print metrics training\n    if ((step+1) // STEPS_PER_EPOCH) > epoch:\n        print(\"\\n Epoch:\", epoch)\n        print('|', end='', flush=True)\n        print(\"TP -  TN  -  FP  -  FN\")\n        print(train_TP.result().numpy(), train_TN.result().numpy(), train_FP.result().numpy(), train_FN.result().numpy())\n        print(\"train AUC: \",train_auc.result().numpy())\n        print(\"train loss: \", train_loss.result().numpy() / STEPS_PER_EPOCH)\n        \n        # validation run for es, it, tr and save model\n        for tokens, labels in valid_dist_ds:\n            strategy.experimental_run_v2(valid_step, args=(tokens, labels))\n            print('=', end='', flush=True)\n\n        # compute metrics\n        print(\"\\n\")\n        print(\"TP -  TN  -  FP  -  FN\")\n        print(valid_TP.result().numpy(), valid_TN.result().numpy(), valid_FP.result().numpy(), valid_FN.result().numpy())\n        print(\"val AUC: \", valid_auc.result().numpy())\n        print(\"val loss: \", valid_loss.result().numpy() / VALIDATION_STEPS)\n\n        # Save predictions and weights of model\n        if (valid_auc.result().numpy() > best_auc) & (epoch>=PREDICT_START_ITER):\n            best_auc = valid_auc.result().numpy()\n            print(\"Prediction on test set - snapshot\")\n            preds = model.predict(test_dataset, verbose = 1)\n            preds_all.append(preds)\n            model.save_weights('best_model.h5') # keep track of best model\n        # set up next epoch\n        epoch = (step+1) // STEPS_PER_EPOCH\n\n        train_auc.reset_states()\n        valid_auc.reset_states()\n\n        valid_loss.reset_states()\n        train_loss.reset_states()\n        \n        train_TP.reset_states()\n        train_TN.reset_states()\n        train_FP.reset_states()\n        train_FN.reset_states()\n        \n        valid_TP.reset_states()\n        valid_TN.reset_states()\n        valid_FP.reset_states()\n        valid_FN.reset_states()\n        \n        if epoch >= EPOCHS:\n            break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Save predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#Generate averages of predictions: last one, and average of snapshots\ntest[\"toxic_best\"] = preds_all[-1]\ntest[\"toxic_avg\"] = sum(preds_all)/len(preds_all)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save the predictions\nMODEL_NAME = MODEL.replace(\"/\", \"-\")\ntest.to_csv(f\"test-{LANG}-{MODEL_NAME}.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stage 2: resume training on validation data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Build datasets objects","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if STAGE2:\n    # the validation set becomes train_dataset\n    train_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices((x_valid, y_valid)) # replaced by x_valid, y_valid!\n        .shuffle(buffer_size=len(x_valid), seed = 18)\n        .prefetch(AUTO)\n        .batch(BATCH_SIZE)\n        .repeat()\n    )\n    \n    # distribute the datset according to the strategy\n    train_dist_ds = strategy.experimental_distribute_dataset(train_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if STAGE2:\n    model.load_weights(\"best_model.h5\") # best model from stage1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Custom training loop","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if STAGE2:\n    STEPS_PER_EPOCH = round(x_valid.shape[0] / (BATCH_SIZE*N_ITER_PER_EPOCH))\n    print(\"Steps per epoch:\", STEPS_PER_EPOCH)\n    EPOCHS = N_EPOCHS*N_ITER_PER_EPOCH\n    best_auc = 0\n    epoch = 0\n\n    preds_all = []\n    for step, (tokens, labels) in enumerate(train_dist_ds):\n        # run training step\n        strategy.experimental_run_v2(train_step, args=(tokens, labels))\n        print('=', end='', flush=True)\n\n        # print metrics training\n        if ((step+1) // STEPS_PER_EPOCH) > epoch:\n            print(\"\\n Epoch:\", epoch)\n            print('|', end='', flush=True)\n            print(\"TP -  TN  -  FP  -  FN\")\n            print(train_TP.result().numpy(), train_TN.result().numpy(), train_FP.result().numpy(), train_FN.result().numpy())\n            print(\"train AUC: \",train_auc.result().numpy())\n            print(\"train loss: \", train_loss.result().numpy() / STEPS_PER_EPOCH)\n\n            # Save predictions and weights of model\n            if epoch>=PREDICT_START_ITER:\n                print(\"Prediction on test set - snapshot\")\n                preds = model.predict(test_dataset, verbose = 1)\n                preds_all.append(preds)\n                \n            # set up next epoch\n            epoch = (step+1) // STEPS_PER_EPOCH\n            \n            train_auc.reset_states()\n            train_loss.reset_states()\n\n            train_TP.reset_states()\n            train_TN.reset_states()\n            train_FP.reset_states()\n            train_FN.reset_states()\n            \n            if epoch >= EPOCHS:\n                # save model if needed\n                model.save_weights('best_model_valid.h5') \n                break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Save predictions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if STAGE2:\n    #Generate averages of snapshot\n    test[\"toxic_mean_snap_valid\"] = sum(preds_all)/len(preds_all)\n    # Save the predictions\n    test.to_csv(f\"test-{LANG}-{MODEL_NAME}.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}