{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Data processing\n\nThis notebook preprocesses our dataset for compatibility with BERT. You should feel free to investigate other solutions (both models and tokenizers)!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os, time\nimport pandas\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom kaggle_datasets import KaggleDatasets\n\n# We'll use a tokenizer for the BERT model from the modelling demo notebook.\n!pip install bert-tensorflow\nimport bert.tokenization\n\nprint(tf.version.VERSION)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Set global variables\n\nSet maximum sequence length and path variables."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"SEQUENCE_LENGTH = 128\n\nDATA_PATH =  \"../input/jigsaw-multilingual-toxic-comment-classification\"\nBERT_PATH = \"../input/bert-multi\"\nBERT_PATH_SAVEDMODEL = os.path.join(BERT_PATH, \"bert_multi_from_tfhub\")\n\nOUTPUT_PATH = \"/kaggle/working\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Examples\n\nLoad and look at examples from [our first competition](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/). These are comments from Wikipedia with a variety of annotations (toxic, obscene, threat, etc)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training data from our first competition,\n# https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data\nwiki_toxic_comment_data = \"jigsaw-toxic-comment-train.csv\"\n\nwiki_toxic_comment_train = pandas.read_csv(os.path.join(\n    DATA_PATH, wiki_toxic_comment_data))\nwiki_toxic_comment_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# BERT Tokenizer\n\nGet the tokenizer corresponding to our multilingual BERT model. See [TensorFlow \nHub](https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/1) for more information about the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_tokenizer(bert_path=BERT_PATH_SAVEDMODEL):\n    \"\"\"Get the tokenizer for a BERT layer.\"\"\"\n    bert_layer = tf.saved_model.load(bert_path)\n    bert_layer = hub.KerasLayer(bert_layer, trainable=False)\n    vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()\n    cased = bert_layer.resolved_object.do_lower_case.numpy()\n    tf.gfile = tf.io.gfile  # for bert.tokenization.load_vocab in tokenizer\n    tokenizer = bert.tokenization.FullTokenizer(vocab_file, cased)\n  \n    return tokenizer\n\ntokenizer = get_tokenizer()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can look at one of our example sentences after we tokenize it, and then again after converting it to word IDs for BERT."},{"metadata":{"trusted":true},"cell_type":"code","source":"example_sentence = wiki_toxic_comment_train.iloc[37].comment_text[:150]\nprint(example_sentence)\n\nexample_tokens = tokenizer.tokenize(example_sentence)\nprint(example_tokens[:17])\n\nexample_input_ids = tokenizer.convert_tokens_to_ids(example_tokens)\nprint(example_input_ids[:17])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing\n\nProcess individual sentences for input to BERT using the tokenizer, and then prepare the entire dataset. The same code will process the other training data files, as well as the validation and test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_sentence(sentence, max_seq_length=SEQUENCE_LENGTH, tokenizer=tokenizer):\n    \"\"\"Helper function to prepare data for BERT. Converts sentence input examples\n    into the form ['input_word_ids', 'input_mask', 'segment_ids'].\"\"\"\n    # Tokenize, and truncate to max_seq_length if necessary.\n    tokens = tokenizer.tokenize(sentence)\n    if len(tokens) > max_seq_length - 2:\n        tokens = tokens[:(max_seq_length - 2)]\n\n    # Convert the tokens in the sentence to word IDs.\n    input_ids = tokenizer.convert_tokens_to_ids([\"[CLS]\"] + tokens + [\"[SEP]\"])\n\n    # The mask has 1 for real tokens and 0 for padding tokens. Only real\n    # tokens are attended to.\n    input_mask = [1] * len(input_ids)\n\n    # Zero-pad up to the sequence length.\n    pad_length = max_seq_length - len(input_ids)\n    input_ids.extend([0] * pad_length)\n    input_mask.extend([0] * pad_length)\n\n    # We only have one input segment.\n    segment_ids = [0] * max_seq_length\n\n    return (input_ids, input_mask, segment_ids)\n\ndef preprocess_and_save_dataset(unprocessed_filename, text_label='comment_text',\n                                seq_length=SEQUENCE_LENGTH, verbose=True):\n    \"\"\"Preprocess a CSV to the expected TF Dataset form for multilingual BERT,\n    and save the result.\"\"\"\n    dataframe = pandas.read_csv(os.path.join(DATA_PATH, unprocessed_filename),\n                                index_col='id')\n    processed_filename = (unprocessed_filename.rstrip('.csv') +\n                          \"-processed-seqlen{}.csv\".format(SEQUENCE_LENGTH))\n\n    pos = 0\n    start = time.time()\n\n    while pos < len(dataframe):\n        processed_df = dataframe[pos:pos + 10000].copy()\n\n        processed_df['input_word_ids'], processed_df['input_mask'], processed_df['all_segment_id'] = (\n            zip(*processed_df[text_label].apply(process_sentence)))\n\n        if pos == 0:\n            processed_df.to_csv(processed_filename, index_label='id', mode='w')\n        else:\n            processed_df.to_csv(processed_filename, index_label='id', mode='a',\n                                header=False)\n\n        if verbose:\n            print('Processed {} examples in {}'.format(\n                pos + 10000, time.time() - start))\n        pos += 10000\n    return\n  \n# Process the training dataset.\npreprocess_and_save_dataset(wiki_toxic_comment_data)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}