{"cells":[{"metadata":{"_uuid":"c5146b549b9069ee005c031080cacc4c297d5ad0"},"cell_type":"markdown","source":"# What if we could finetune BERT from Apache MXNet?"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"\nThis Kernel shows how one can fine-tune BERT Pretrained model on the Quora dataset using [Apache MXNet](https://mxnet.incubator.apache.org/) and [GluonNLP](http://gluon-nlp.mxnet.io/). \n\nRules of the competition doesn't allow to use pretrained models, and that's why I haven't submitted results of my training. But If I run it in my EC2 using the following command line:\n\n`python train.py --gpu 1,2 --processed_train_data ../processed/train.data --processed_dev_data ../processed/dev.data --processed_word_vocab ../processed/word_vocab.data --batch_size 50 --output submission_multilingual.txt`\n\nI am receiving the following results, which looks higher than current leaderboard values (it takes about 2 hours per epoch):\n\n`\nEpoch 0. Current Loss: 0.13936. Val F1: 0.681, Threshold: 0.480\nWed Jan 16 08:43:40 2019\nEpoch 1. Current Loss: 0.13253. Val F1: 0.713, Threshold: 0.360\nWed Jan 16 10:52:28 2019\nEpoch 2. Current Loss: 0.12653. Val F1: 0.725, Threshold: 0.370\nWed Jan 16 13:00:41 2019\nEpoch 3. Current Loss: 0.12500. Val F1: 0.748, Threshold: 0.500\nWed Jan 16 15:08:15 2019\nEpoch 4. Current Loss: 0.12243. Val F1: 0.775, Threshold: 0.480\nWed Jan 16 17:15:47 2019\n`\n\nFeel free to explore the code and use for tuning your submissions.\n\nUnfortunately, some of the code hasn't been released in the latest version of GLuonNLP.  For being able to create a stand alone example, I have copied some code into here as is. \n\nLinks to original files included, so one could remove the implementation from here and use original one once new release is made.\nThe name of the file in the first lines of the cells below is the name of original file in `src/` folder of my project."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# bert_adam.py - BERT authors did a change to Adam algorightm for better convergence\n# taken from: https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/optimizer/bert_adam.py\n\n# coding: utf-8\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n\"\"\"Weight updating functions.\"\"\"\nfrom mxnet.optimizer import Optimizer, register\nfrom mxnet.ndarray import zeros, NDArray\n\n__all__ = ['BERTAdam']\n\n@register\nclass BERTAdam(Optimizer):\n    \"\"\"The Adam optimizer with weight decay regularization for BERT.\n\n    Updates are applied by::\n\n        rescaled_grad = clip(grad * rescale_grad, clip_gradient)\n        m = beta1 * m + (1 - beta1) * rescaled_grad\n        v = beta2 * v + (1 - beta2) * (rescaled_grad**2)\n        w = w - learning_rate * (m / (sqrt(v) + epsilon) + wd * w)\n\n    Note that this is different from `mxnet.optimizer.Adam`, where L2 loss is added and\n    accumulated in m and v. In BERTAdam, the weight decay term decoupled from gradient\n    based update.\n\n    This is also slightly different from the AdamW optimizer described in\n    *Fixing Weight Decay Regularization in Adam*, where the schedule multiplier and\n    learning rate is decoupled, and the bias-correction terms are removed.\n    The BERTAdam optimizer uses the same learning rate to apply gradients\n    w.r.t. the loss and weight decay.\n\n    This optimizer accepts the following parameters in addition to those accepted\n    by :class:`mxnet.optimizer.Optimizer`.\n\n    Parameters\n    ----------\n    beta1 : float, optional\n        Exponential decay rate for the first moment estimates.\n    beta2 : float, optional\n        Exponential decay rate for the second moment estimates.\n    epsilon : float, optional\n        Small value to avoid division by 0.\n    \"\"\"\n    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8,\n                 **kwargs):\n        super(BERTAdam, self).__init__(learning_rate=learning_rate, **kwargs)\n        self.beta1 = beta1\n        self.beta2 = beta2\n        self.epsilon = epsilon\n\n    def create_state(self, index, weight): # pylint: disable=unused-argument\n        \"\"\"Initialization for mean and var.\"\"\"\n        return (zeros(weight.shape, weight.context, dtype=weight.dtype), #mean\n                zeros(weight.shape, weight.context, dtype=weight.dtype)) #variance\n\n    def update(self, index, weight, grad, state):\n        \"\"\"Update method.\"\"\"\n        try:\n            from mxnet.ndarray.contrib import adamw_update\n        except ImportError:\n            raise ImportError('Failed to import nd.contrib.adamw_update from MXNet. '\n                              'BERTAdam optimizer requires mxnet>=1.5.0b20181228. '\n                              'Please upgrade your MXNet version.')\n        assert(isinstance(weight, NDArray))\n        assert(isinstance(grad, NDArray))\n        self._update_count(index)\n        lr = self._get_lr(index)\n        wd = self._get_wd(index)\n\n        kwargs = {'beta1': self.beta1, 'beta2': self.beta2, 'epsilon': self.epsilon,\n                  'rescale_grad': self.rescale_grad}\n        if self.clip_gradient:\n            kwargs['clip_gradient'] = self.clip_gradient\n\n        mean, var = state\n        adamw_update(weight, grad, mean, var, out=weight, lr=1, wd=wd, eta=lr, **kwargs)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c3b8b2ab643edc7f09d6a77e0d568c34bf5b291"},"cell_type":"code","source":"# bert_tokenizer.py - bunch of classes need to run tokenization for BERT\n# Taken from https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/tokenizer.py\n\n# coding=utf-8\n# Copyright 2018 The Google AI Language Team Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Tokenization classes.\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport collections\nimport unicodedata\nimport io\nimport six\n\ndef convert_to_unicode(text):\n    \"\"\"Converts `text` to Unicode (if it's not already), assuming utf-8 input.\"\"\"\n    if six.PY3:\n        if isinstance(text, str):\n            return text\n        elif isinstance(text, bytes):\n            return text.decode('utf-8', 'ignore')\n        else:\n            raise ValueError('Unsupported string type: %s' % (type(text)))\n    elif six.PY2:\n        if isinstance(text, str):\n            return text.decode('utf-8', 'ignore')\n        elif isinstance(text, unicode):  # noqa: F821\n            return text\n        else:\n            raise ValueError('Unsupported string type: %s' % (type(text)))\n    else:\n        raise ValueError('Not running on Python2 or Python 3?')\n\n\ndef printable_text(text):\n    \"\"\"Returns text encoded in a way suitable for print.\"\"\"\n    # These functions want `str` for both Python2 and Python3, but in one case\n    # it's a Unicode string and in the other it's a byte string.\n    if six.PY3:\n        if isinstance(text, str):\n            return text\n        elif isinstance(text, bytes):\n            return text.decode('utf-8', 'ignore')\n        else:\n            raise ValueError('Unsupported string type: %s' % (type(text)))\n    elif six.PY2:\n        if isinstance(text, str):\n            return text\n        elif isinstance(text, unicode):  # noqa: F821\n            return text.encode('utf-8')\n        else:\n            raise ValueError('Unsupported string type: %s' % (type(text)))\n    else:\n        raise ValueError('Not running on Python2 or Python 3?')\n\n\ndef load_vocab(vocab_file):\n    \"\"\"Loads a vocabulary file into a dictionary.\"\"\"\n    vocab = collections.OrderedDict()\n    index = 0\n    with io.open(vocab_file, 'r') as reader:\n        while True:\n            token = convert_to_unicode(reader.readline())\n            if not token:\n                break\n            token = token.strip()\n            vocab[token] = index\n            index += 1\n    return vocab\n\n\ndef convert_tokens_to_ids(vocab, tokens):\n    \"\"\"Converts a sequence of tokens into ids using the vocab.\"\"\"\n    ids = []\n    for token in tokens:\n        ids.append(vocab[token])\n    return ids\n\n\ndef whitespace_tokenize(text):\n    \"\"\"Runs basic whitespace cleaning and splitting on a piece of text.\"\"\"\n    text = text.strip()\n    if not text:\n        return []\n    tokens = text.split()\n    return tokens\n\n\nclass FullTokenizer(object):\n    \"\"\"Runs end-to-end tokenziation.\n\n    Parameters\n    ----------\n    vocab : Vocab\n        Vocabulary for the corpus.\n    do_lower_case : bool, default True\n        Convert tokens to lower cases.\n    \"\"\"\n\n    def __init__(self, vocab, do_lower_case=True):\n        self.vocab = vocab\n        self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)\n        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)\n\n    def tokenize(self, text):\n        split_tokens = []\n        for token in self.basic_tokenizer.tokenize(text):\n            for sub_token in self.wordpiece_tokenizer.tokenize(token):\n                split_tokens.append(sub_token)\n\n        return split_tokens\n\n    def convert_tokens_to_ids(self, tokens):\n        return convert_tokens_to_ids(self.vocab, tokens)\n\n\nclass BasicTokenizer(object):\n    \"\"\"Runs basic tokenization (punctuation splitting, lower casing, etc.).\"\"\"\n\n    def __init__(self, do_lower_case=True):\n        \"\"\"Constructs a BasicTokenizer.\n\n        Args:\n          do_lower_case: Whether to lower case the input.\n        \"\"\"\n        self.do_lower_case = do_lower_case\n\n    def tokenize(self, text):\n        \"\"\"Tokenizes a piece of text.\"\"\"\n        text = convert_to_unicode(text)\n        text = self._clean_text(text)\n\n        # This was added on November 1st, 2018 for the multilingual and Chinese\n        # models. This is also applied to the English models now, but it doesn't\n        # matter since the English models were not trained on any Chinese data\n        # and generally don't have any Chinese data in them (there are Chinese\n        # characters in the vocabulary because Wikipedia does have some Chinese\n        # words in the English Wikipedia.).\n        text = self._tokenize_chinese_chars(text)\n        orig_tokens = whitespace_tokenize(text)\n        split_tokens = []\n        for token in orig_tokens:\n            if self.do_lower_case:\n                token = token.lower()\n                token = self._run_strip_accents(token)\n            split_tokens.extend(self._run_split_on_punc(token))\n\n        output_tokens = whitespace_tokenize(' '.join(split_tokens))\n        return output_tokens\n\n    def _run_strip_accents(self, text):\n        \"\"\"Strips accents from a piece of text.\"\"\"\n        text = unicodedata.normalize('NFD', text)\n        output = []\n        for char in text:\n            cat = unicodedata.category(char)\n            if cat == 'Mn':\n                continue\n            output.append(char)\n        return ''.join(output)\n\n    def _run_split_on_punc(self, text):\n        \"\"\"Splits punctuation on a piece of text.\"\"\"\n        chars = list(text)\n        i = 0\n        start_new_word = True\n        output = []\n        while i < len(chars):\n            char = chars[i]\n            if _is_punctuation(char):\n                output.append([char])\n                start_new_word = True\n            else:\n                if start_new_word:\n                    output.append([])\n                start_new_word = False\n                output[-1].append(char)\n            i += 1\n\n        return [''.join(x) for x in output]\n\n    def _tokenize_chinese_chars(self, text):\n        \"\"\"Adds whitespace around any CJK character.\"\"\"\n        output = []\n        for char in text:\n            cp = ord(char)\n            if self._is_chinese_char(cp):\n                output.append(' ')\n                output.append(char)\n                output.append(' ')\n            else:\n                output.append(char)\n        return ''.join(output)\n\n    def _is_chinese_char(self, cp):\n        \"\"\"Checks whether CP is the codepoint of a CJK character.\"\"\"\n        # This defines a \"chinese character\" as anything in the CJK Unicode block:\n        #   https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)\n        #\n        # Note that the CJK Unicode block is NOT all Japanese and Korean characters,\n        # despite its name. The modern Korean Hangul alphabet is a different block,\n        # as is Japanese Hiragana and Katakana. Those alphabets are used to write\n        # space-separated words, so they are not treated specially and handled\n        # like the all of the other languages.\n        if ((cp >= 0x4E00 and cp <= 0x9FFF)\n                or (cp >= 0x3400 and cp <= 0x4DBF)\n                or (cp >= 0x20000 and cp <= 0x2A6DF)\n                or (cp >= 0x2A700 and cp <= 0x2B73F)\n                or (cp >= 0x2B740 and cp <= 0x2B81F)\n                or (cp >= 0x2B820 and cp <= 0x2CEAF)\n                or (cp >= 0xF900 and cp <= 0xFAFF)\n                or (cp >= 0x2F800 and cp <= 0x2FA1F)):\n            return True\n\n        return False\n\n    def _clean_text(self, text):\n        \"\"\"Performs invalid character removal and whitespace cleanup on text.\"\"\"\n        output = []\n        for char in text:\n            cp = ord(char)\n            if cp == 0 or cp == 0xfffd or _is_control(char):\n                continue\n            if _is_whitespace(char):\n                output.append(' ')\n            else:\n                output.append(char)\n        return ''.join(output)\n\n\nclass WordpieceTokenizer(object):\n    \"\"\"Runs WordPiece tokenziation.\"\"\"\n\n    def __init__(self, vocab, unk_token='[UNK]', max_input_chars_per_word=100):\n        self.vocab = vocab\n        self.unk_token = unk_token\n        self.max_input_chars_per_word = max_input_chars_per_word\n\n    def tokenize(self, text):\n        \"\"\"Tokenizes a piece of text into its word pieces.\n\n        This uses a greedy longest-match-first algorithm to perform tokenization\n        using the given vocabulary.\n\n        For example:\n          input = \"unaffable\"\n          output = [\"un\", \"##aff\", \"##able\"]\n\n        Args:\n          text: A single token or whitespace separated tokens. This should have\n            already been passed through `BasicTokenizer.\n\n        Returns:\n          A list of wordpiece tokens.\n        \"\"\"\n\n        text = convert_to_unicode(text)\n\n        output_tokens = []\n        for token in whitespace_tokenize(text):\n            chars = list(token)\n            if len(chars) > self.max_input_chars_per_word:\n                output_tokens.append(self.unk_token)\n                continue\n\n            is_bad = False\n            start = 0\n            sub_tokens = []\n            while start < len(chars):\n                end = len(chars)\n                cur_substr = None\n                while start < end:\n                    substr = ''.join(chars[start:end])\n                    if start > 0:\n                        substr = '##' + substr\n                    if substr in self.vocab:\n                        cur_substr = substr\n                        break\n                    end -= 1\n                if cur_substr is None:\n                    is_bad = True\n                    break\n                sub_tokens.append(cur_substr)\n                start = end\n\n            if is_bad:\n                output_tokens.append(self.unk_token)\n            else:\n                output_tokens.extend(sub_tokens)\n        return output_tokens\n\n\ndef _is_whitespace(char):\n    \"\"\"Checks whether `chars` is a whitespace character.\"\"\"\n    # \\t, \\n, and \\r are technically control characters but we treat them\n    # as whitespace since they are generally considered as such.\n    if char in [' ', '\\t', '\\n', '\\r']:\n        return True\n    cat = unicodedata.category(char)\n    if cat == 'Zs':\n        return True\n    return False\n\n\ndef _is_control(char):\n    \"\"\"Checks whether `chars` is a control character.\"\"\"\n    # These are technically control characters but we count them as whitespace\n    # characters.\n    if char in ['\\t', '\\n', '\\r']:\n        return False\n    cat = unicodedata.category(char)\n    if cat.startswith('C'):\n        return True\n    return False\n\n\ndef _is_punctuation(char):\n    \"\"\"Checks whether `chars` is a punctuation character.\"\"\"\n    cp = ord(char)\n    # We treat all non-letter/number ASCII as punctuation.\n    # Characters such as \"^\", \"$\", and \"`\" are not in the Unicode\n    # Punctuation class but we treat them as punctuation anyways, for\n    # consistency.\n    group0 = cp >= 33 and cp <= 47\n    group1 = cp >= 58 and cp <= 64\n    group2 = cp >= 91 and cp <= 96\n    group3 = cp >= 123 and cp <= 126\n    if (group0 or group1 or group2 or group3):\n        return True\n    cat = unicodedata.category(char)\n    if cat.startswith('P'):\n        return True\n    return False\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9dae3fc773f251ecb6744602c445e529fe797fe"},"cell_type":"code","source":"# bert_transform.py - Class incapsulates data transformation logic for Classification.\n# Taken from: https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/dataset.py\n\n# coding=utf-8\n# Copyright 2018 The Google AI Language Team Authors and DMLC.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport numpy as np\nimport six\n\n\ndef _truncate_seq_pair(tokens_a, tokens_b, max_length):\n    \"\"\"Truncates a sequence pair in place to the maximum length.\"\"\"\n    # This is a simple heuristic which will always truncate the longer sequence\n    # one token at a time. This makes more sense than truncating an equal percent\n    # of tokens from each, since if one sequence is very short then each token\n    # that's truncated likely contains more information than a longer sequence.\n    while True:\n        total_length = len(tokens_a) + len(tokens_b)\n        if total_length <= max_length:\n            break\n        if len(tokens_a) > len(tokens_b):\n            tokens_a.pop()\n        else:\n            tokens_b.pop()\n\n\ndef convert_to_unicode(text):\n    \"\"\"Converts `text` to Unicode (if it's not already), assuming utf-8 input.\"\"\"\n    if six.PY3:\n        if isinstance(text, str):\n            return text\n        elif isinstance(text, bytes):\n            return text.decode('utf-8', 'ignore')\n        else:\n            raise ValueError('Unsupported string type: %s' % (type(text)))\n    elif six.PY2:\n        if isinstance(text, str):\n            return text.decode('utf-8', 'ignore')\n        elif isinstance(text, unicode):  # noqa: F821\n            return text\n        else:\n            raise ValueError('Unsupported string type: %s' % (type(text)))\n    else:\n        raise ValueError('Not running on Python2 or Python 3?')\n\n\nclass BERTTransform(object):\n    \"\"\"BERT style data transformation.\n\n    Parameters\n    ----------\n    tokenizer : BasicTokenizer or FullTokensizer.\n        Tokenizer for the sentences.\n    max_seq_length : int.\n        Maximum sequence length of the sentences.\n    pad : bool, default True\n        Whether to pad the sentences to maximum length.\n    pair : bool, default True\n        Whether to transform sentences or sentence pairs.\n    \"\"\"\n    def __init__(self, tokenizer, max_seq_length, pad=True, pair=True):\n        self._tokenizer = tokenizer\n        self._max_seq_length = max_seq_length\n        self._pad = pad\n        self._pair = pair\n\n    def __call__(self, line):\n        \"\"\"Perform transformation for sequence pairs or single sequences.\n\n        The transformation is processed in the following steps:\n        - tokenize the input sequences\n        - insert [CLS], [SEP] as necessary\n        - generate type ids to indicate whether a token belongs to the first\n          sequence or the second sequence.\n        - generate valid length\n\n        For sequence pairs, the input is a tuple of 2 strings:\n        text_a, text_b.\n\n        Inputs:\n            text_a: 'is this jacksonville ?'\n            text_b: 'no it is not'\n        Tokenization:\n            text_a: 'is this jack ##son ##ville ?'\n            text_b: 'no it is not .'\n        Processed:\n            tokens:  '[CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]'\n            type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1\n            valid_length: 14\n\n        For single sequences, the input is a tuple of single string: text_a.\n        Inputs:\n            text_a: 'the dog is hairy .'\n        Tokenization:\n            text_a: 'the dog is hairy .'\n        Processed:\n            text_a:  '[CLS] the dog is hairy . [SEP]'\n            type_ids: 0     0   0   0  0     0 0\n            valid_length: 7\n\n        Parameters\n        ----------\n        line: tuple of str\n            Input strings. For sequence pairs, the input is a tuple of 3 strings:\n            (text_a, text_b). For single sequences, the input is a tuple of single\n            string: (text_a,).\n\n        Returns\n        -------\n        np.array: input token ids in 'int32', shape (batch_size, seq_length)\n        np.array: valid length in 'int32', shape (batch_size,)\n        np.array: input token type ids in 'int32', shape (batch_size, seq_length)\n        \"\"\"\n        # convert to unicode\n        text_a = line[0]\n        text_a = convert_to_unicode(text_a)\n        if self._pair:\n            assert len(line) == 2\n            text_b = line[1]\n            text_b = convert_to_unicode(text_b)\n\n        tokens_a = self._tokenizer.tokenize(text_a)\n        tokens_b = None\n\n        if self._pair:\n            tokens_b = self._tokenizer.tokenize(text_b)\n\n        if tokens_b:\n            # Modifies `tokens_a` and `tokens_b` in place so that the total\n            # length is less than the specified length.\n            # Account for [CLS], [SEP], [SEP] with \"- 3\"\n            _truncate_seq_pair(tokens_a, tokens_b, self._max_seq_length - 3)\n        else:\n            # Account for [CLS] and [SEP] with \"- 2\"\n            if len(tokens_a) > self._max_seq_length - 2:\n                tokens_a = tokens_a[0:(self._max_seq_length - 2)]\n\n        # The embedding vectors for `type=0` and `type=1` were learned during\n        # pre-training and are added to the wordpiece embedding vector\n        # (and position vector). This is not *strictly* necessary since\n        # the [SEP] token unambiguously separates the sequences, but it makes\n        # it easier for the model to learn the concept of sequences.\n\n        # For classification tasks, the first vector (corresponding to [CLS]) is\n        # used as as the \"sentence vector\". Note that this only makes sense because\n        # the entire model is fine-tuned.\n        tokens = []\n        segment_ids = []\n        tokens.append('[CLS]')\n        segment_ids.append(0)\n        for token in tokens_a:\n            tokens.append(token)\n            segment_ids.append(0)\n        tokens.append('[SEP]')\n        segment_ids.append(0)\n\n        if tokens_b:\n            for token in tokens_b:\n                tokens.append(token)\n                segment_ids.append(1)\n            tokens.append('[SEP]')\n            segment_ids.append(1)\n\n        input_ids = self._tokenizer.convert_tokens_to_ids(tokens)\n\n        # The valid length of sentences. Only real  tokens are attended to.\n        valid_length = len(input_ids)\n\n        if self._pad:\n            # Zero-pad up to the sequence length.\n            padding_length = self._max_seq_length - valid_length\n            # use padding tokens for the rest\n            input_ids.extend([self._tokenizer.vocab['[PAD]']] * padding_length)\n            segment_ids.extend([self._tokenizer.vocab['[PAD]']] * padding_length)\n\n        return np.array(input_ids, dtype='int32'), np.array(valid_length, dtype='int32'),\\\n               np.array(segment_ids, dtype='int32')\n\n\nclass ClassificationTransform(object):\n    \"\"\"Dataset Transformation for BERT-style Sentence Classification.\n\n    Parameters\n    ----------\n    tokenizer : BasicTokenizer or FullTokensizer.\n        Tokenizer for the sentences.\n    labels : list of int.\n        List of all label ids for the classification task.\n    max_seq_length : int.\n        Maximum sequence length of the sentences.\n    pad : bool, default True\n        Whether to pad the sentences to maximum length.\n    pair : bool, default True\n        Whether to transform sentences or sentence pairs.\n    \"\"\"\n    def __init__(self, tokenizer, labels, max_seq_length, pad=True, pair=True):\n        self._label_map = {}\n        for (i, label) in enumerate(labels):\n            self._label_map[label] = i\n        self._bert_xform = BERTTransform(tokenizer, max_seq_length, pad=pad, pair=pair)\n\n    def __call__(self, line):\n        \"\"\"Perform transformation for sequence pairs or single sequences.\n\n        The transformation is processed in the following steps:\n        - tokenize the input sequences\n        - insert [CLS], [SEP] as necessary\n        - generate type ids to indicate whether a token belongs to the first\n          sequence or the second sequence.\n        - generate valid length\n\n        For sequence pairs, the input is a tuple of 3 strings:\n        text_a, text_b and label.\n\n        Inputs:\n            text_a: 'is this jacksonville ?'\n            text_b: 'no it is not'\n            label: '0'\n        Tokenization:\n            text_a: 'is this jack ##son ##ville ?'\n            text_b: 'no it is not .'\n        Processed:\n            tokens:  '[CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]'\n            type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1\n            valid_length: 14\n            label: 0\n\n        For single sequences, the input is a tuple of 2 strings: text_a and label.\n        Inputs:\n            text_a: 'the dog is hairy .'\n            label: '1'\n        Tokenization:\n            text_a: 'the dog is hairy .'\n        Processed:\n            text_a:  '[CLS] the dog is hairy . [SEP]'\n            type_ids: 0     0   0   0  0     0 0\n            valid_length: 7\n            label: 1\n\n        Parameters\n        ----------\n        line: tuple of str\n            Input strings. For sequence pairs, the input is a tuple of 3 strings:\n            (text_a, text_b, label). For single sequences, the input is a tuple\n            of 2 strings: (text_a, label).\n\n        Returns\n        -------\n        np.array: input token ids in 'int32', shape (batch_size, seq_length)\n        np.array: valid length in 'int32', shape (batch_size,)\n        np.array: input token type ids in 'int32', shape (batch_size, seq_length)\n        np.array: label id in 'int32', shape (batch_size, 1)\n        \"\"\"\n        label = line[-1]\n        label = convert_to_unicode(label)\n        label_id = self._label_map[label]\n        label_id = np.array([label_id], dtype='int32')\n        input_ids, valid_length, segment_ids = self._bert_xform(line[:-1])\n        return input_ids, valid_length, segment_ids, label_id","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25c95c2861785f019b740900498cadeba0d55e53"},"cell_type":"code","source":"# data_pipeline.py - Dataset for Quora data with a few helper methods.\n\nimport os\nimport csv\nfrom mxnet.gluon.data import ArrayDataset\n\n\ndef get_sub_segment_from_list(dataset, indices):\n    return ArrayDataset([dataset[i] for i in indices])\n\n\nclass QuoraDataset(ArrayDataset):\n    \"\"\"This dataset provides access to Quora insincere data competition\"\"\"\n    def __init__(self, segment, root_dir=\"../input/\"):\n        self._root_dir = root_dir\n        self._segment = segment\n        self._segments = {\n            'train': 'train.csv',\n            'test': 'test.csv'\n        }\n\n        super(QuoraDataset, self).__init__(self._read_data())\n\n    def _read_data(self):\n        file_path = os.path.join(self._root_dir, self._segments[self._segment])\n        with open(file_path, mode='r', encoding='utf-8', newline='') as f:\n            reader = csv.reader(f, delimiter=',', quotechar='\"')\n            # ignore 1st line - which is header\n            data = [(i, ) + tuple(row) for i, row in enumerate(reader) if i > 0]\n\n        return data\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c7928a72f6e4e0e9900b7f30780d93540288ac8b"},"cell_type":"code","source":"# model.py - Fine-tuned mode, that using BERT as the base.\nfrom mxnet.gluon.nn import Dense, Dropout, HybridSequential, Block\n\n\nclass QuoraModel(Block):\n    def __init__(self, bert, dropout=0.1, prefix=None, params=None):\n        super(QuoraModel, self).__init__(prefix=prefix, params=params)\n\n        self.bert = bert\n\n        with self.name_scope():\n            self.output = HybridSequential()\n\n            with self.output.name_scope():\n                if dropout:\n                    self.output.add(Dropout(dropout))\n                self.output.add(Dense(units=1))\n\n    def forward(self, inputs, token_types, valid_length=None):  # pylint: disable=arguments-differ\n        _, pooler_out = self.bert(inputs, token_types, valid_length)\n        return self.output(pooler_out)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c0fd44e9cc0863b7b2622ce778ac4c8781b8962"},"cell_type":"code","source":"# train.py - main training and evaluation logic using everything on top.\n\nimport os\n\nimport argparse\nimport time\n\nimport pandas as pd\nimport numpy as np\nimport pickle\nfrom gluonnlp.data import FixedBucketSampler\nfrom gluonnlp.data.batchify import Tuple, Stack\nfrom gluonnlp.model import get_model\n\nimport mxnet as mx\nfrom mxnet import nd, autograd, gluon\nfrom mxnet.gluon import Trainer\nfrom mxnet.gluon.data import DataLoader, SimpleDataset\nfrom mxnet.gluon.utils import clip_global_norm\n\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import train_test_split\n\n# Uncomment this if copy files locally\n# from src.bert_adam import BERTAdam\n# from src.bert_tokenizer import FullTokenizer\n# from src.bert_transform import ClassificationTransform\n# from src.data_pipeline import QuoraDataset, get_sub_segment_from_list\n# from src.model import QuoraModel\n\n\ndef parse_args():\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--gpu', type=str, default=None, help='Index of GPU to use')\n    parser.add_argument('--processed_train_data', type=str, default=None,\n                        help='Path to processed training data')\n    parser.add_argument('--processed_dev_data', type=str, default=None,\n                        help='Path to processed val data')\n    parser.add_argument('--processed_word_vocab', type=str, default=None,\n                        help='Path to processed word level vocab data')\n    parser.add_argument('--logging_path', default=\"./log.txt\",\n                        help='logging file path')\n    parser.add_argument('--model_path', default='./model',\n                        help='saving model in model_path')\n    parser.add_argument('--epochs', type=int, default=5, help='training epochs')\n    parser.add_argument('--batch_size', type=int, default=100, help='batch size')\n    parser.add_argument('--num_buckets', type=int, default=10, help='Buckets')\n    parser.add_argument('--dropout', type=float, default=0, help='Dropout rate for everything')\n    parser.add_argument('--warmup_ratio', type=float, default=0.1,\n                        help='ratio of warmup steps used in NOAM\\'s stepsize schedule')\n    parser.add_argument('--lr', type=float, default=5e-6, help='Initial learning rate')\n    parser.add_argument('--log_interval', type=int, default=200, help='Initial learning rate')\n    parser.add_argument('--threshold', type=float, default=0.4, help='Positive/negative class threshold')\n    parser.add_argument('--output', type=str, default='submission.txt', help='Submission file')\n    parser.add_argument('--clip', type=int, default=1, help='Gradient clipping value')\n    parser.add_argument('--sw_0_class', type=float, default=0.7, help='Sample weight for 0 class')\n    parser.add_argument('--sw_1_class', type=float, default=0.5, help='Sample weight for 1 class (will be added to sample weight of 0 class)')\n    return parser.parse_args()\n\n\ndef find_best_f1(outputs, labels):\n    tmp = [0, 0, 0]  # idx, cur, max\n    threshold = 0\n\n    for tmp[0] in np.arange(0.1, 0.501, 0.01):\n        tmp[1] = f1_score(labels.asnumpy(), outputs.asnumpy() > tmp[0])\n        if tmp[1] > tmp[2]:\n            threshold = tmp[0]\n            tmp[2] = tmp[1]\n\n    return tmp[2], threshold\n\n\ndef filter_out_question_id(*items):\n    \"\"\"Method makes dataset DataLoadable\"\"\"\n    # q_idx, q_id, input_ids, valid_lengths, type_ids, label\n    if len(items) == 6:\n        return items[0], items[2], items[3], items[4], items[5]\n    else:\n        return items[0], items[2], items[3], items[4]\n\n\ndef run_training(net, trainer, train_dataloader, val_dataloader,\n                 epochs, clip, sw_0_class, sw_1_class, log_interval, threshold, context):\n    loss_fn = mx.gluon.loss.SigmoidBinaryCrossEntropyLoss(from_sigmoid=False)\n    print(\"Start training for {} epochs: {}\".format(epochs, time.ctime()))\n\n    differentiable_params = [p for p in net.collect_params().values() if p.grad_req != 'null']\n\n    for e in range(epochs):\n        train_loss = 0.\n        total_items = 0\n        val_outputs = []\n        val_l = []\n\n        for step_num, (q_idx, input_ids, valid_lengths, type_ids, label) in enumerate(train_dataloader):\n            items_per_iteration = q_idx.shape[0]\n            total_items += items_per_iteration\n\n            input_ids = gluon.utils.split_and_load(input_ids, context, even_split=False)\n            valid_lengths = gluon.utils.split_and_load(valid_lengths.astype('float32'), context, even_split=False)\n            type_ids = gluon.utils.split_and_load(type_ids, context, even_split=False)\n            label = gluon.utils.split_and_load(label, context, even_split=False)\n\n            losses = []\n\n            for i, (id, vl, ti, l) in enumerate(zip(input_ids, valid_lengths, type_ids, label)):\n                with autograd.record():\n                    out = net(id, ti, vl)\n                    # penalize for class 1 more than for class 0\n                    sample_weights = (sw_1_class * l.astype('float32')) + sw_0_class\n                    loss = loss_fn(out, l.astype('float32'), sample_weights)\n                    losses.append(loss)\n\n            for loss in losses:\n                loss.backward()\n\n            grads = [p.grad(context[0]) for p in differentiable_params]\n            clip_global_norm(grads, clip)\n\n            if len(context) > 1:\n                # in multi gpu mode we propagate new gradients to the rest of gpus\n                for p in differentiable_params:\n                    grads = p.list_grad()\n                    source = grads[0]\n                    destination = grads[1:]\n\n                    for dest in destination:\n                        source.copyto(dest)\n\n            trainer.step(1)\n\n            for loss in losses:\n                train_loss += loss.sum().asscalar()\n\n            # Uncomment this, if you want to display info after every iteration\n            # print('[Epoch {} Batch {}/{}] loss={:.4f}, lr={:.7f}'\n            #       .format(e, step_num + 1, len(train_dataloader),\n            #               train_loss / total_items,\n            #               trainer.learning_rate))\n\n        nd.waitall()\n        # print(\"Epoch training finished at {}.\".format(time.ctime()))\n\n        for step_num, (q_idx, input_ids, valid_lengths, type_ids, label) in enumerate(val_dataloader):\n            input_ids = gluon.utils.split_and_load(input_ids, context, even_split=False)\n            valid_lengths = gluon.utils.split_and_load(valid_lengths.astype('float32'), context, even_split=False)\n            type_ids = gluon.utils.split_and_load(type_ids, context, even_split=False)\n            label = gluon.utils.split_and_load(label, context, even_split=False)\n\n            for i, (id, vl, ti, l) in enumerate(zip(input_ids, valid_lengths, type_ids, label)):\n                out = net(id, ti, vl)\n                val_outputs.append(out.sigmoid().as_in_context(mx.cpu()))\n                val_l.append(l.reshape(shape=(l.shape[0], 1)).as_in_context(mx.cpu()))\n\n        val_outputs = mx.nd.concat(*val_outputs, dim=0)\n        val_l = mx.nd.concat(*val_l, dim=0)\n\n        val_f1, threshold = find_best_f1(val_outputs, val_l)\n        nd.waitall()\n\n        print(\"Epoch {}. Current Loss: {:.5f}. Val F1: {:.3f}, Threshold: {:.3f}\"\n              .format(e, train_loss / total_items, val_f1, threshold))\n        print(time.ctime())\n\n    return net, threshold\n\n\ndef run_evaluation(net, dataloader, threshold):\n    print(\"Start predicting\")\n    outputs = []\n    result = []\n\n    # label here is added by BERTTransformer - it is fake and always 0\n    for step_num, (q_idx, input_ids, valid_lengths, type_ids, label) in enumerate(dataloader):\n        q_idx = gluon.utils.split_and_load(q_idx, context, even_split=False)\n        input_ids = gluon.utils.split_and_load(input_ids, context, even_split=False)\n        valid_lengths = gluon.utils.split_and_load(valid_lengths.astype('float32'), context,\n                                                   even_split=False)\n        type_ids = gluon.utils.split_and_load(type_ids, context, even_split=False)\n\n        for i, (qid, id, vl, ti) in enumerate(zip(q_idx, input_ids, valid_lengths, type_ids)):\n            out = net(id, ti, vl)\n            outputs.append((qid.as_in_context(mx.cpu()),\n                            out.sigmoid().as_in_context(mx.cpu()) > threshold))\n\n    for batch in outputs:\n        result.extend([(int(q_idx.asscalar()), int(pred.asscalar()))\n                       for q_idx, pred in zip(batch[0], batch[1])])\n\n    return result\n\n\ndef load_and_process_dataset(dataset, word_vocab, path=None):\n    if path is not None and os.path.exists(path):\n        processed_dataset = pickle.load(open(path, 'rb'))\n        return processed_dataset\n\n    tokenizer = FullTokenizer(word_vocab, do_lower_case=True)\n    transformer = ClassificationTransform(tokenizer, ['0', '1'], max_seq_length=74, pair=False)\n    # test data doesn't have label - use 0\n    processed_dataset = SimpleDataset([(item[0], item[1],) +\n                                       transformer((item[2], item[3] if len(item) >= 4 else '0'))\n                                       for item in dataset])\n\n    if path:\n        pickle.dump(processed_dataset, open(path, 'wb'))\n\n    return processed_dataset\n\n\ndef main():\n    args = parse_args()\n    context = [mx.cpu(0)] if args.gpu is None else [mx.gpu(int(i)) for i in args.gpu.split(',')]\n\n    # Loading pretrained BERT model and Vocab\n    bert, word_vocab = get_model('bert_12_768_12',\n                                 dataset_name='wiki_multilingual',\n                                 pretrained=True,\n                                 use_pooler=True,\n                                 use_decoder=False,\n                                 use_classifier=False,\n                                 ctx=context)\n\n    model = QuoraModel(bert, dropout=args.dropout)\n    model.output.initialize(init=mx.init.Normal(0.02), ctx=context)\n\n    dataset = QuoraDataset('train')\n\n    train_indices, dev_indices = train_test_split(range(len(dataset)), train_size=0.98)\n\n    train_dataset = load_and_process_dataset(get_sub_segment_from_list(dataset, train_indices),\n                                             word_vocab,\n                                             args.processed_train_data)\n\n    dev_dataset = load_and_process_dataset(get_sub_segment_from_list(dataset, dev_indices),\n                                           word_vocab,\n                                           args.processed_dev_data)\n\n    batchify_fn = Tuple(Stack(),\n                        Stack(),\n                        Stack(),\n                        Stack(),\n                        Stack())\n\n    train_sampler = FixedBucketSampler(lengths=[item[3] for item in train_dataset],\n                                       batch_size=args.batch_size,\n                                       shuffle=True,\n                                       num_buckets=args.num_buckets)\n\n    dev_sampler = FixedBucketSampler(lengths=[item[3] for item in dev_dataset],\n                                     batch_size=args.batch_size,\n                                     shuffle=True)\n\n    train_dataloader = DataLoader(train_dataset.transform(filter_out_question_id),\n                                  batchify_fn=batchify_fn,\n                                  batch_sampler=train_sampler,\n                                  num_workers=10)\n\n    dev_dataloader = DataLoader(dev_dataset.transform(filter_out_question_id),\n                                batchify_fn=batchify_fn,\n                                batch_sampler=dev_sampler,\n                                num_workers=5)\n\n    optimizer = BERTAdam(learning_rate=args.lr)\n    trainer = Trainer(model.collect_params(), optimizer=optimizer)\n\n    best_model, best_threshold = run_training(model, trainer, train_dataloader, dev_dataloader,\n                                              args.epochs, args.clip, args.sw_0_class,\n                                              args.sw_1_class, args.log_interval, args.threshold,\n                                              context)\n\n    test_dataset = QuoraDataset('test')\n    processed_test_dataset = load_and_process_dataset(test_dataset, word_vocab)\n\n    batchify_test_fn = Tuple(Stack(),\n                             Stack(),\n                             Stack(),\n                             Stack(),\n                             Stack())\n\n    test_sampler = FixedBucketSampler(lengths=[item[3] for item in processed_test_dataset],\n                                      batch_size=args.batch_size)\n\n    test_dataloader = DataLoader(processed_test_dataset.transform(filter_out_question_id),\n                                 batchify_fn=batchify_test_fn,\n                                 batch_sampler=test_sampler,\n                                 num_workers=10,\n                                 shuffle=False)\n\n    predictions = run_evaluation(model, test_dataloader, best_threshold)\n\n    submission = pd.DataFrame()\n    mapping = {item[0]: item[1] for item in test_dataset}\n    submission['qid'] = [mapping[p[0]] for p in predictions]\n    submission['prediction'] = [p[1] for p in predictions]\n\n    submission.to_csv(args.output, index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64f298e8c0d7def75390c08b6c5bf07e19709167"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"797d7a7249d5744a79d29110661f5bb6f87c6d33"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}