{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Competition\n\n* [Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)* is the 3rd annual competition organized by the Jigsaw team.\n*  It follows *[Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)*, the original 2018 competition, \n* and *[Jigsaw Unintended Bias in Toxicity Classification](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification)*, which required the competitors to consider biased ML predictions in their new models.\n* In 2020, the goal is :\n        * to use english only training data \n        * and to run toxicity predictions on many different languages which can be done using **multilingual models**, and **speed up using TPUs.**\n\n## About this notebook\nMany awesome notebooks has already been made so far. Many of them used really cool technologies like [Pytorch XLA](https://www.kaggle.com/theoviel/bert-pytorch-huggingface-starter). This notebook instead aims at constructing a **fast, concise, reusable, and beginner-friendly model scaffold**. It will focus on the following points:\n* **Using Tensorflow and Keras**: Tensorflow is a powerful framework, and Keras makes the training process extremely easy to understand. This is especially good for beginners to learn how to use TPUs, and for experts to focus on the modelling aspect.\n* **Using Huggingface's `transformers` library**: [This library](https://huggingface.co/transformers/) is extremely popular, so using this let you easily integrate the end result into your ML pipelines, and can be easily reused for your other projects.\n* **Multilingual DistilBERT**: DistilBERT is a faster and lighter version of BERT; this model let you quickly experiments with different ideas, all while achieving\n* **Native TPU usage**: The TPU usage is abstracted using the native `strategy` that was created using Tensorflow's `tf.distribute.experimental.TPUStrategy`. This avoids getting too much into the lower-level aspect of TPU management.\n* **Use a subset of the data**: Instead of using the entire dataset, we will only use the 2018 subset of the data available, which makes this much faster, all while achieving a respectable accuracy."},{"metadata":{},"cell_type":"markdown","source":"## **Data**\n\n* The primary data for the competition is, in each provided file, the comment_text column. This contains the text of a comment which has been classified as toxic or non-toxic (0...1 in the toxic column). The **train set’s comments are entirely in english** and come either from **Civil Comments** or **Wikipedia** talk page edits. The **test data**'s comment_text columns are **composed of multiple non-English languages.**\n\n* The *-train.csv files and validation.csv file also contain a toxic column that is the target to be trained on.\n\n* The **jigsaw-toxic-comment-train.csv** and **jigsaw-unintended-bias-train.csv** contain training data (comment_text and toxic) from the **two previous Jigsaw competitions**, as well as additional columns that you may find useful."},{"metadata":{},"cell_type":"markdown","source":"## Files\n* jigsaw-toxic-comment-train.csv - data from our first competition. The dataset is made up of English comments from Wikipedia’s talk page edits.\n* jigsaw-unintended-bias-train.csv - data from our second competition. This is an expanded version of the Civil Comments dataset with a range of additional labels.\n* sample_submission.csv - a sample submission file in the correct format\n* test.csv - comments from Wikipedia talk pages in different non-English languages.\n* validation.csv - comments from Wikipedia talk pages in different non-English languages.\n* jigsaw-toxic-comment-train-processed-seqlen128.csv - training data preprocessed for BERT\n* jigsaw-unintended-bias-train-processed-seqlen128.csv - training data preprocessed for BERT\n* validation-processed-seqlen128.csv - validation data preprocessed for BERT\n* test-processed-seqlen128.csv - test data preprocessed for BERT"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nimport transformers\nfrom tqdm.notebook import tqdm\nfrom tokenizers import BertWordPieceTokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n    tokenizer.enable_truncation(max_length=maxlen)\n    tokenizer.enable_padding(max_length=maxlen)\n    all_ids = []\n    \n    for i in tqdm(range(0, len(texts), chunk_size)):\n        text_chunk = texts[i:i+chunk_size].tolist()\n        encs = tokenizer.encode_batch(text_chunk)\n        all_ids.extend([enc.ids for enc in encs])\n    \n    return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, max_len=512):\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=1e-5), loss='binary_crossentropy', metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Configs"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Data access\nGCS_DS_PATH = KaggleDatasets().get_gcs_path()\n\n# Configuration\nEPOCHS = 2\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create fast tokenizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n\n# Save the loaded tokenizer locally\nsave_path = '/kaggle/working/distilbert_base_uncased/'\nif not os.path.exists(save_path):\n    os.makedirs(save_path)\ntokenizer.save_pretrained(save_path)\n\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('distilbert_base_uncased/vocab.txt', lowercase=True)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load text data into memory"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -al /kaggle/input/jigsaw-multilingual-toxic-comment-classification","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Engl comments from Wikipedia’s talk page\ntrain1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\n# Civil Comments dataset with a range of additionnal labels.\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\n# comments from Wikipedia talk pages in different non-English languages.                     \nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv') \ntest_no_labels = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train1.iloc[:4]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Valid and test labeled data building"},{"metadata":{"trusted":true},"cell_type":"code","source":"#test_no_labels : not used in this notebook because only used for Kaggle model evaluation only \ntest_no_labels[0:4]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid.iloc[:4]\nprint ('valid shape:',valid.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Evenly split valid data to build new valid ndtest labeled data\nvalid_with_labels = valid[0:3999]\ntest_with_labels = valid[4000:7999]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = fast_encode(train1.comment_text.astype(str), fast_tokenizer, maxlen=512)\nx_valid = fast_encode(valid_with_labels.comment_text.astype(str), fast_tokenizer, maxlen=512)\nx_test = fast_encode(test_with_labels.comment_text.astype(str), fast_tokenizer, maxlen=512)\n\ny_train = train1.toxic.values\ny_valid = valid_with_labels.toxic.values\ny_test = test_with_labels.toxic.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build datasets objects"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(BATCH_SIZE)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_test, y_test))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n   ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load model into the TPU"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = transformers.TFDistilBertModel.from_pretrained('distilbert-base-multilingual-cased')\n    model = build_model(transformer_layer, max_len=512)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/working","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.chdir(r'/kaggle/working')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Working_Out = '/kaggle/working'\n\n#Save best weight model\nModelPath = Working_Out + '/weights_best_inception3_pool_over1.hdf5'\ncheckpointer = ModelCheckpoint(filepath = ModelPath, verbose=1, save_best_only=True)\n\nhistory = model.fit(\n    train_dataset,\n    steps_per_epoch=N_STEPS,\n    validation_data=valid_dataset,\n    epochs=5,\n    callbacks=[checkpointer]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the training process \ntrain_acc = history.history['accuracy']\nval_acc = history.history['val_accuracy'] \n\ntrain_loss = history.history['loss']\nval_loss = history.history['val_loss']\n \nplt.plot(train_loss, label=['loss'])\nplt.plot(val_loss, label=['val_loss'])\nplt.title('Loss evolution at each epochs')\nplt.legend()\nplt.show()\n\nplt.plot(train_acc , label=['accuracy'])\nplt.plot(val_acc , label=['val_accuracy'])\nplt.title('Accuracy evolution at each epochs')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('\\n# Evaluate')\nresult = model.evaluate(test_dataset)\ndict(zip(model.metrics_names, result))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}