{"cells":[{"metadata":{},"cell_type":"markdown","source":"**To run this sample on Google Cloud Platform with various accelerator setups:**\n\n 1. Download this notebook\n 1. Create a Cloud AI Platform Notebook VM with your choice of accelerator.\n   * V100 GPU ([AI Platform Notebook UI](https://console.cloud.google.com/ai-platform/notebooks) > New Instance > Tensorflow 2.2 > Customize > V100 x1)\n   * 4x V100 GPU ([AI Platform Notebook UI](https://console.cloud.google.com/ai-platform/notebooks) > New Instance > Tensorflow 2.2 > Customize > V100 x 4)\n   * 8x V100 GPU ([AI Platform Notebook UI](https://console.cloud.google.com/ai-platform/notebooks) > New Instance > Tensorflow 2.2 > Customize > V100 x 8)\n   * TPU v3-8 (use `create-tpu-deep-learning-vm.sh` script from [this page](create-tpu-deep-learning-vm.sh) with `--tpu-type v3-8`)\n   * TPU v3-32 pod (use `create-tpu-deep-learning-vm.sh` script from [this page](create-tpu-deep-learning-vm.sh) with `--tpu-type v3-32`)\n 1. Get the data from Kaggle. The easiest is to run the cell below on Kaggle and copy the name of the GCS bucket where the dataset is cached. This bucket is a cache and will expire after a couple of days but it should be enough to run the notebook. Optionnally, for best performance, copy the data to your own bucket located in the same region as your TPU.\n 1. adjust the import and the `GCS_PATH` in the cell below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# When not running on Kaggle, comment out this import\nfrom kaggle_datasets import KaggleDatasets\n# When not running on Kaggle, set a fixed GCS path here\nGCS_PATH = KaggleDatasets().get_gcs_path('jigsaw-multilingual-toxic-comment-classification')\nprint(GCS_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f2ef93cd-2bc2-4681-bc6e-9a9395b25691","_uuid":"ebd8119d-fb40-454f-a46f-940e80e7f8ce"},"cell_type":"markdown","source":"# Overview\n\nThis notebook is a fork of the Geting started notebook for the [Jigsaw Multilingual Toxic Comment classification competition](https://www.kaggle.com/kivlichangoogle/jigsaw-multilingual-getting-started) by [Ian Kivlichan](https://www.kaggle.com/kivlichangoogle).\n\nIt only takes one toxic comment to sour an online discussion. The Conversation AI team, a research initiative founded by [Jigsaw](https://jigsaw.google.com/) and Google, builds technology to protect voices in conversation. A main area of focus is machine learning models that can identify toxicity in online conversations, where toxicity is defined as anything *rude, disrespectful or otherwise likely to make someone leave a discussion*. Our API, [Perspective](http://perspectiveapi.com/), serves these models and others in a growing set of languages (see our [documentation](https://github.com/conversationai/perspectiveapi/blob/master/2-api/models.md#all-attribute-types) for the full list). If these toxic contributions can be identified, we could have a safer, more collaborative internet.\n\nIn this competition, we'll explore how models for recognizing toxicity in online conversations might generalize across different languages. Specifically, in this notebook, we'll demonstrate this with a multilingual BERT (m-BERT) model. Multilingual BERT is pretrained on monolingual data in a variety of languages, and through this learns multilingual representations of text. These multilingual representations enable *zero-shot cross-lingual transfer*, that is, by fine-tuning on a task in one language, m-BERT can learn to perform that same task in another language (for some examples, see e.g. [How multilingual is Multilingual BERT?](https://arxiv.org/abs/1906.01502)).\n\nWe'll study this zero-shot transfer in the context of toxicity in online conversations, similar to past competitions we've hosted ([[1]](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification), [[2]](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)). But rather than analyzing toxicity in English as in those competitions, here we'll ask you to do it in several different languages. For training, we're including the (English) datasets from our earlier competitions, as well as a small amount of new toxicity data in other languages."},{"metadata":{"_cell_guid":"593619be-8090-4dad-a462-e883e560ec1c","_uuid":"cba509cb-708d-4fd0-8e2a-e5c1c7a0982d","trusted":true},"cell_type":"code","source":"import os, time, logging\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom matplotlib import pyplot as plt\nprint(tf.version.VERSION)\ntf.get_logger().setLevel(logging.ERROR)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ae9fd8d4-bb00-4b3d-8452-0ec605f6ebba","_uuid":"add2f478-e634-42db-b3c6-732ebd484ce3"},"cell_type":"markdown","source":"# TPU or GPU detection"},{"metadata":{"_cell_guid":"f5ccaf08-c532-4fde-9306-b897c890d0f8","_uuid":"bc97f110-17eb-44dd-a792-d66c27a0b3a6","trusted":true},"cell_type":"code","source":"try: # detect TPU\n    tpu = None\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() # TPU detection\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept ValueError: # detect GPU(s) and enable mixed precision\n    strategy = tf.distribute.MirroredStrategy() # works on GPU and multi-GPU\n    policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n    tf.config.optimizer.set_jit(True) # XLA compilation\n    tf.keras.mixed_precision.experimental.set_policy(policy)\n    print('Mixed precision enabled')\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\n# mixed precision\n# On TPU, bfloat16/float32 mixed precision is automatically used in TPU computations.\n# Enabling it in Keras also stores relevant variables in bfloat16 format (memory optimization).\n# This additional optimization was not used for TPUs in this sample.\n# On GPU, specifically V100, mixed precision must be enabled for hardware TensorCores to be used.\n# XLA compilation must be enabled for this to work. (On TPU, XLA compilation is the default and cannot be turned off)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c55830a8-39e7-4607-9691-cd0848a092f7","_uuid":"b1188d07-d23d-4d04-b9f5-125d16b982ed"},"cell_type":"markdown","source":"# Configuration\nSet maximum sequence length and path variables."},{"metadata":{"_cell_guid":"7acc953d-b1f6-4d29-a006-69af6839f7a9","_uuid":"7d8d39c7-1fd2-4af7-a88a-9ca1242a6277","trusted":true},"cell_type":"code","source":"SEQUENCE_LENGTH = 128\n\n# Copy of the TF Hub model at https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/2\nBERT_GCS_PATH = 'gs://bert_multilingual_public/bert_multi_cased_L-12_H-768_A-12_2/'\nEPOCHS = 6\n\nif tpu:\n    BATCH_SIZE = 128 * strategy.num_replicas_in_sync\nelse:\n    BATCH_SIZE = 64 * strategy.num_replicas_in_sync\n\nTRAIN_DATA = GCS_PATH + \"/jigsaw-toxic-comment-train-processed-seqlen{}.csv\".format(SEQUENCE_LENGTH)\nTRAIN_DATA_LENGTH = 223549 # rows\nVALID_DATA = GCS_PATH + \"/validation-processed-seqlen{}.csv\".format(SEQUENCE_LENGTH)\nSTEPS_PER_EPOCH = TRAIN_DATA_LENGTH // BATCH_SIZE\n\nLR_MAX = 0.001 * strategy.num_replicas_in_sync\nLR_EXP_DECAY = .9\nLR_MIN = 0.0001\n\n@tf.function\ndef lr_fn(epoch):\n    lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch) + LR_MIN\n    return lr\n\nprint(\"Learning rate schedule:\")\nrng = [i for i in range(EPOCHS)]\ny = [lr_fn(x) for x in rng]\nplt.plot(rng, [lr_fn(x) for x in rng])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model\n\nDefine the model. We convert m-BERT's output to a final probabilty estimate. We're using an [m-BERT model from TensorFlow Hub](https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/1)."},{"metadata":{"_cell_guid":"fcd2093b-0774-4adf-9e4c-e9096f156d32","_uuid":"1392a5e0-c8e4-46ea-b45d-0d9289682e09","trusted":true},"cell_type":"code","source":"def multilingual_bert_model(max_seq_length=SEQUENCE_LENGTH):\n    \"\"\"Build and return a multilingual BERT model and tokenizer.\"\"\"\n    input_word_ids = tf.keras.layers.Input(\n        shape=(max_seq_length,), dtype=tf.int32, name=\"input_word_ids\")\n    input_mask = tf.keras.layers.Input(\n        shape=(max_seq_length,), dtype=tf.int32, name=\"input_mask\")\n    segment_ids = tf.keras.layers.Input(\n        shape=(max_seq_length,), dtype=tf.int32, name=\"all_segment_id\")\n    \n    bert_layer = tf.saved_model.load(BERT_GCS_PATH)  # copy of TF Hub model 'https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/2'\n    bert_layer = hub.KerasLayer(bert_layer, trainable=True)\n\n    pooled_output, _ = bert_layer([input_word_ids, input_mask, segment_ids])\n    output = tf.keras.layers.Dense(32, activation='relu')(pooled_output)\n    output = tf.keras.layers.Dense(1, activation='sigmoid', name='labels', dtype=tf.float32)(output)\n\n    return tf.keras.Model(inputs={'input_word_ids': input_word_ids,\n                                  'input_mask': input_mask,\n                                  'all_segment_id': segment_ids},\n                          outputs=output)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"48479a32-25c1-40c9-bd1c-d076eb39d86e","_uuid":"b24c47ad-a156-41f2-97b8-f5618181382c"},"cell_type":"markdown","source":"# Dataset\nLoad the preprocessed dataset. See the demo notebook for sample code for performing this preprocessing."},{"metadata":{"trusted":true},"cell_type":"code","source":"def parse_string_list_into_ints(strlist):\n    s = tf.strings.strip(strlist)\n    s = tf.strings.substr(\n        strlist, 1, tf.strings.length(s) - 2)  # Remove parentheses around list\n    s = tf.strings.split(s, ',', maxsplit=SEQUENCE_LENGTH)\n    s = tf.strings.to_number(s, tf.int32)\n    s = tf.reshape(s, [SEQUENCE_LENGTH])  # Force shape here needed for XLA compilation (TPU)\n    return s\n\ndef format_sentences(data, label='toxic', remove_language=False):\n    labels = {'labels': data.pop(label)}\n    if remove_language:\n        languages = {'language': data.pop('lang')}\n    # The remaining three items in the dict parsed from the CSV are lists of integers\n    for k,v in data.items():  # \"input_word_ids\", \"input_mask\", \"all_segment_id\"\n        data[k] = parse_string_list_into_ints(v)\n    return data, labels\n\ndef make_sentence_dataset_from_csv(filename, label='toxic', language_to_filter=None):\n    # This assumes the column order label, input_word_ids, input_mask, segment_ids\n    SELECTED_COLUMNS = [label, \"input_word_ids\", \"input_mask\", \"all_segment_id\"]\n    label_default = tf.int32 if label == 'id' else tf.float32\n    COLUMN_DEFAULTS  = [label_default, tf.string, tf.string, tf.string]\n\n    if language_to_filter:\n        insert_pos = 0 if label != 'id' else 1\n        SELECTED_COLUMNS.insert(insert_pos, 'lang')\n        COLUMN_DEFAULTS.insert(insert_pos, tf.string)\n\n    preprocessed_sentences_dataset = tf.data.experimental.make_csv_dataset(\n        filename, column_defaults=COLUMN_DEFAULTS, select_columns=SELECTED_COLUMNS,\n        batch_size=1, num_epochs=1, shuffle=False)  # We'll do repeating and shuffling ourselves\n    # make_csv_dataset required a batch size, but we want to batch later\n    preprocessed_sentences_dataset = preprocessed_sentences_dataset.unbatch()\n    \n    if language_to_filter:\n        preprocessed_sentences_dataset = preprocessed_sentences_dataset.filter(\n            lambda data: tf.math.equal(data['lang'], tf.constant(language_to_filter)))\n        #preprocessed_sentences.pop('lang')\n    preprocessed_sentences_dataset = preprocessed_sentences_dataset.map(\n        lambda data: format_sentences(data, label=label,\n                                      remove_language=language_to_filter))\n\n    return preprocessed_sentences_dataset\n\ndef count_dataset_steps(dataset):\n    # to be used on small datasets only: iterates through entire dataset and counts\n    cnt = 0\n    for data in dataset:\n        cnt += 1\n    return cnt","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e006f30f-6350-45d5-b452-338c4bc78cc5","_uuid":"10ff216a-2248-4104-858c-2b2461b42fba"},"cell_type":"markdown","source":"Set up our data pipelines for training and evaluation."},{"metadata":{"_cell_guid":"b17b24d7-1b0f-442b-bfd2-c42748bcd067","_uuid":"864f31ac-8285-442b-93cf-dcae11d7fe62","trusted":true},"cell_type":"code","source":"def make_dataset_pipeline(dataset, repeat_and_shuffle=True):\n    \"\"\"Set up the pipeline for the given dataset.\n    \n    Caches, repeats, shuffles, and sets the pipeline up to prefetch batches.\"\"\"\n    cached_dataset = dataset.cache()\n    if repeat_and_shuffle:\n        cached_dataset = cached_dataset.repeat().shuffle(2048)\n        cached_dataset = cached_dataset.batch(BATCH_SIZE, drop_remainder=True) # no remainder on repeated dataset\n    else:\n        cached_dataset = cached_dataset.batch(BATCH_SIZE)\n    cached_dataset = cached_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n    return cached_dataset\n\n# Load the preprocessed English dataframe.\npreprocessed_en_filename = TRAIN_DATA\n\n# Set up the dataset and pipeline.\nenglish_train_dataset = make_dataset_pipeline(\n    make_sentence_dataset_from_csv(preprocessed_en_filename))\n\n# Process the new datasets by language.\npreprocessed_val_filename = VALID_DATA\n\nnonenglish_val_datasets = {}\nnonenglish_val_datasets_steps = {}\nfor language_name, language_label in [('Spanish', 'es'), ('Italian', 'it'),\n                                      ('Turkish', 'tr')]:\n    nonenglish_val_datasets[language_name] = make_sentence_dataset_from_csv(\n        preprocessed_val_filename, language_to_filter=language_label)\n    nonenglish_val_datasets[language_name] = make_dataset_pipeline(\n        nonenglish_val_datasets[language_name], repeat_and_shuffle=False)\n    nonenglish_val_datasets_steps[language_name] = count_dataset_steps(nonenglish_val_datasets[language_name])\n\nnonenglish_val_datasets['Combined'] = make_sentence_dataset_from_csv(preprocessed_val_filename)\nnonenglish_val_datasets['Combined'] = make_dataset_pipeline(nonenglish_val_datasets['Combined'], repeat_and_shuffle=False)\nnonenglish_val_datasets_steps['Combined'] = count_dataset_steps(nonenglish_val_datasets['Combined'])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ffd5c9ef-a806-4ae6-a1c5-8723ed822232","_uuid":"0779be08-0502-47c0-b284-5f3d851de2e1"},"cell_type":"markdown","source":"# Instantiate the model\n\nCompile our model. We will fine-tune the multilingual model on one of our English datasets, and then evaluate its performance on the new multilingual toxicity data. As our metric, we'll use the [AUC](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC)."},{"metadata":{"_cell_guid":"422a984e-e571-4898-9667-b95d38416ddd","_uuid":"e3d569ca-0bf4-4bde-aa69-95a46908f65a","trusted":true},"cell_type":"code","source":"with strategy.scope():\n    multilingual_bert = multilingual_bert_model()\n\n    # Compile the model. Optimize using stochastic gradient descent.\n    multilingual_bert.compile(\n        loss=tf.keras.losses.BinaryCrossentropy(),\n        optimizer=tf.keras.optimizers.SGD(learning_rate=0.001*strategy.num_replicas_in_sync),\n        metrics=[tf.keras.metrics.AUC()],\n        steps_per_execution=16)\n\nmultilingual_bert.summary()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f477d7f4-20ec-4858-87d6-c041313ad276","_uuid":"730e2b4b-6e7c-43f3-912e-3cbea3db62c2","trusted":true},"cell_type":"code","source":"%%time\n# Train on English Wikipedia comment data.\nlr_callback = tf.keras.callbacks.LearningRateScheduler(lr_fn)\nhistory = multilingual_bert.fit(\n    english_train_dataset, steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS,\n    #validation_data=nonenglish_val_datasets['Combined'], validation_steps=nonenglish_val_datasets_steps['Combined'],\n    callbacks=[lr_callback])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Performance on non-English comments after training.\nfor language in nonenglish_val_datasets:\n    results = multilingual_bert.evaluate(nonenglish_val_datasets[language],\n                                         steps=nonenglish_val_datasets_steps[language], verbose=0)\n    print('{} loss, AUC after training:'.format(language), results)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# License"},{"metadata":{},"cell_type":"markdown","source":"Copyright 2020 Google LLC\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n\n---\n\n\nThis is not an official Google product but sample code provided for an educational purpose\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}