{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# About this notebook\n\nThe goal of this notebook is experimenting with a cross-lingual text classifier based on transfer learning and Bayesian neural networks. The [Jigsaw Multilingual Toxic Comment Classification challenge](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification) is selected as data source for labeled cross-lingual texts. This task has two nuances:\n\n1. I want to build a semantic space which is independent from a concrete language.\n\n2. I have a large labeled text corpus (hundreds of thousands labeled texts) for only single language, but training datasets for other languages are very small or thay are empty at all. And I have to classify texts just in such languages.\n\nA transfer learning helps us to account the first of these nuances. I use a pre-trained [XLM-RoBERTa](https://arxiv.org/abs/1911.02116) model as initial state for a Siamese neural network which trains to transform a common semantic space into its special analogue, into which a distance between tonally opposed texts is larger and distance between texts with same sentiments is smaller. For learning of the Siamese NN, I apply special loss function which is known as [Distance Based Logistic Loss (a DBL loss)](https://arxiv.org/abs/1608.00161). Such loss is better than usual cross-entropy loss, because it is contrastive-based, and any contrastive-based loss guarantees that the Siamese neural network after its training will calculate a compact space with required semantic properties. In comparison with a \"classical\" contrastive loss, which is popular for Siamese neural networks, the DBL loss is more effective owing to quicker convergence. It is known, that a Triplet Loss is also used for the Siamese neural network, but in my experiments with textual data this loss didn't allow well separable semantic space, and so I didn't include it in this notebook.\n\nAfter the transformation of semantic space using the Siamese XLM-RoBERTa, I have to build a final classifier, which becomes to use the resultant space as a feature vector generator for toxic/normal classification of texts in various languages, which are different from the base language of text pairs used for the Siamese NN training. I have a large labeled text corpus in English (our base language), but I need to recognize texts in French, Portuguese, Russian, Spanish, Italian, and Turkish, and a little labeled data are available only for the last three languages. In this case, I use [a Bayesian neural network](https://arxiv.org/abs/1505.05424) as a final classifier. All weights of neurons in this network are stochastic. Such neural networks efficiently model an epistemic uncertainty, produced by a lack of knowledge about the modeled process (toxicity in different natural languages). In this notebook, I use the Tensorflow-Probability implementation of the Bayesian neural network which is based on the [Flipout method](https://arxiv.org/abs/1803.04386). Also, I apply the AutoML approach which is based on Bayesian optimization: a number of hidden layers (i.e. depth of neural network) and KL weight are selected to maximize the Bayesian neural network quality on the cross-validated data. At that, the cross-validation is based on splitting by three abovementioned languages (so, it is not random splitting), and this approach helps to better estimate unbiased quality for any unknown language.\n\nFurthermore, I have to describe yet some nuance about KL weight. As you see formulation (8) of loss function for the Bayesian neural network in the [corresponded paper](https://arxiv.org/abs/1505.05424), this loss function consists of two items: \"traditional\" logarithm of the likelihood function (a data-dependent part) and \"bayesian\" Kullback-Leibler (KL) divergence between distribution on the weights q(w|θ) and the true Bayesian posterior on the weights (a prior-dependent part). And the \"bayesian\" item (KL divergence) can be interpreted as the complexity cost, so it is a powerful regularization term in the loss function. In practice, the regularization term must be balanced using special weight, which can be named as KL weight in this case. So, by analogy with the [sklearn's LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html), I introduce a positive floating-point parameter `C` as an inverse of regularization strength (abovementioned KL weight), and smaller values of this parameter specify stronger regularization.","metadata":{}},{"cell_type":"code","source":"import codecs\nimport copy\nimport csv\nimport gc\nimport os\nimport pickle\nimport random\nimport tempfile\nimport time\nfrom typing import Dict, List, Sequence, Set, Tuple, Union","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom matplotlib import figure\nimport numpy as np\nimport seaborn as sns\nfrom scipy.stats import hmean\nfrom sklearn.manifold import TSNE\nfrom sklearn.metrics import roc_auc_score, roc_curve, average_precision_score\nfrom skopt import gp_minimize\nfrom skopt.plots import plot_convergence, plot_objective, plot_evaluations\nfrom skopt.space import Real, Integer\nfrom skopt.utils import use_named_args\nimport tensorflow as tf\nimport tensorflow_probability as tfp\nfrom tensorflow.python.framework import ops, tensor_util\nfrom tensorflow.python.keras.utils import losses_utils, tf_utils\nfrom tensorflow.python.ops import math_ops\nfrom tensorflow.python.ops.losses import util as tf_losses_util\nimport tensorflow_addons as tfa\nfrom transformers import AutoTokenizer, XLMRobertaTokenizer\nfrom transformers import TFXLMRobertaModel, XLMRobertaConfig","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class LossFunctionWrapper(tf.keras.losses.Loss):\n    def __init__(self,\n                 fn,\n                 reduction=losses_utils.ReductionV2.AUTO,\n                 name=None,\n                 **kwargs):\n        super(LossFunctionWrapper, self).__init__(reduction=reduction, name=name)\n        self.fn = fn\n        self._fn_kwargs = kwargs\n\n    def call(self, y_true, y_pred):\n        if tensor_util.is_tensor(y_pred) and tensor_util.is_tensor(y_true):\n            y_pred, y_true = tf_losses_util.squeeze_or_expand_dimensions(y_pred, y_true)\n        return self.fn(y_true, y_pred, **self._fn_kwargs)\n\n    def get_config(self):\n        config = {}\n        for k, v in six.iteritems(self._fn_kwargs):\n            config[k] = tf.keras.backend.eval(v) if tf_utils.is_tensor_or_variable(v) \\\n                else v\n        base_config = super(LossFunctionWrapper, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def distance_based_log_loss(y_true, y_pred):\n    y_pred = ops.convert_to_tensor(y_pred)\n    y_true = math_ops.cast(y_true, y_pred.dtype)\n    margin = 1.0\n    p = (1.0 + tf.math.exp(-margin)) / (1.0 + tf.math.exp(y_pred - margin))\n    return tf.keras.backend.binary_crossentropy(target=y_true, output=p)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DBLLogLoss(LossFunctionWrapper):\n    def __init__(self, reduction=losses_utils.ReductionV2.AUTO,\n                 name='distance_based_log_loss'):\n        super(DBLLogLoss, self).__init__(distance_based_log_loss, name=name,\n                                         reduction=reduction)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MaskCalculator(tf.keras.layers.Layer):\n    def __init__(self, output_dim, **kwargs):\n        self.output_dim = output_dim\n        super(MaskCalculator, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        super(MaskCalculator, self).build(input_shape)\n\n    def call(self, inputs, **kwargs):\n        return tf.keras.backend.permute_dimensions(\n            x=tf.keras.backend.repeat(\n                x=tf.keras.backend.cast(\n                    x=tf.keras.backend.greater(\n                        x=inputs,\n                        y=0\n                    ),\n                    dtype='float32'\n                ),\n                n=self.output_dim\n            ),\n            pattern=(0, 2, 1)\n        )\n\n    def compute_output_shape(self, input_shape):\n        assert len(input_shape) == 1\n        shape = list(input_shape)\n        shape.append(self.output_dim)\n        return tuple(shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class WeightPosteriorCallback(tf.keras.callbacks.Callback):\n    def __init__(self, log_every: int):\n        self.bayesian_layers = []\n        self.layer_names = []\n        self.logged_epochs = dict()\n        self.log_every = log_every\n        super(WeightPosteriorCallback, self).__init__()\n    \n    def on_train_begin(self, logs=None):\n        self.bayesian_layers = list(filter(\n            lambda layer: (layer.name.lower().startswith('bayesianhiddenlayer') or \\\n                           layer.name.lower().startswith('bayesianoutputlayer')),\n            self.model.layers\n        ))\n        self.layer_names = []\n        max_layer_idx = 0\n        for layer in self.bayesian_layers:\n            if layer.name.lower().startswith('bayesianhiddenlayer'):\n                new_layer_name = 'Layer'\n                start_pos = len('bayesianhiddenlayer')\n                find_idx = layer.name[start_pos:].find('_')\n                assert find_idx > 0\n                layer_idx = int(layer.name[start_pos:(start_pos + find_idx)])\n                new_layer_name += '{0}'.format(layer_idx)\n                if layer_idx > max_layer_idx:\n                    max_layer_idx = layer_idx\n            else:\n                new_layer_name = 'Layer{0}'.format(max_layer_idx + 1)\n            self.layer_names.append(new_layer_name)\n        self.logged_epochs = dict()\n    \n    def on_epoch_end(self, epoch, logs=None):\n        if (epoch == 0) or (((epoch + 1) % self.log_every) == 0):\n            qm_vals = [layer.kernel_posterior.mean() for layer in self.bayesian_layers]\n            qs_vals = [layer.kernel_posterior.stddev() for layer in self.bayesian_layers]\n            self.logged_epochs[epoch + 1] = (qm_vals, qs_vals)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_weight_posteriors(layer_names: List[str],\n                           logged_epochs: Dict[int, Tuple[List[np.ndarray], List[np.ndarray]]]):\n    epoch_indices = sorted(list(logged_epochs.keys()))\n    assert len(epoch_indices) > 0\n    n_layers = len(layer_names)\n    assert n_layers > 0\n    for epoch_idx in epoch_indices:\n        assert len(logged_epochs[epoch_idx][0]) == len(logged_epochs[epoch_idx][1])\n        assert len(logged_epochs[epoch_idx][0]) == n_layers\n    fig = plt.figure(figsize=(12, 6 * len(epoch_indices)))\n    counter = 1\n    for epoch_idx in epoch_indices:\n        qm_vals = logged_epochs[epoch_idx][0]\n        qs_vals = logged_epochs[epoch_idx][1]\n\n        ax = fig.add_subplot(len(epoch_indices), 2, counter)\n        for n, qm in zip(layer_names, qm_vals):\n            sns.distplot(tf.reshape(qm, shape=[-1]), ax=ax, label=n)\n        ax.set_title('Epoch {0}, weight means'.format(epoch_idx))\n        ax.set_xlim([-1.5, 1.5])\n        ax.legend(loc='best')\n        counter += 1\n\n        ax = fig.add_subplot(len(epoch_indices), 2, counter)\n        for n, qs in zip(layer_names, qs_vals):\n            sns.distplot(tf.reshape(qs, shape=[-1]), ax=ax)\n        ax.set_title('Epoch {0}, weight stddevs'.format(epoch_idx))\n        ax.set_xlim([0, 1.])\n        counter += 1\n\n    fig.tight_layout()\n    plt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def generate_random_seed() -> int:\n    return random.randint(0, 2147483648)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def regular_encode(texts: List[str], tokenizer: XLMRobertaTokenizer,\n                   maxlen: int) -> Tuple[np.ndarray, np.ndarray]:\n    err_msg = '\"{0}\" is wrong type for the text list!'.format(type(texts))\n    assert isinstance(texts, list) or isinstance(texts, tuple), err_msg\n    enc_di = tokenizer.batch_encode_plus(\n        texts,\n        return_token_type_ids=False,\n        padding='max_length',\n        max_length=maxlen\n    )\n    err_msg = '{0} != {1}'.format(len(texts), len(enc_di['input_ids']))\n    assert len(texts) == len(enc_di['input_ids']), err_msg\n    err_msg = '{0} != {1}'.format(len(texts), len(enc_di['attention_mask']))\n    assert len(texts) == len(enc_di['attention_mask']), err_msg\n    encoded_tokens = np.zeros((len(texts), maxlen), dtype=np.int32)\n    encoded_masks = np.zeros((len(texts), maxlen), dtype=np.int32)\n    for sample_idx, (encoded_cur_text, encoded_cur_mask) in enumerate(\n        zip(enc_di['input_ids'], enc_di['attention_mask'])\n    ):\n        n_text = len(encoded_cur_text)\n        n_mask = len(encoded_cur_mask)\n        err_msg = 'Tokens and masks of texts \"{0}\" are different! '\\\n                  '{1} != {2}'.format(texts[sample_idx], n_text, n_mask)\n        assert n_text == n_mask, err_msg\n        if n_text >= maxlen:\n            encoded_tokens[sample_idx] = np.array(encoded_cur_text[0:maxlen],\n                                                  dtype=np.int32)\n            encoded_masks[sample_idx] = np.array(encoded_cur_mask[0:maxlen],\n                                                 dtype=np.int32)\n        else:\n            padding = [0 for _ in range(maxlen - n_text)]\n            encoded_tokens[sample_idx] = np.array(encoded_cur_text + padding,\n                                                  dtype=np.int32)\n            encoded_masks[sample_idx] = np.array(encoded_cur_mask + padding,\n                                                 dtype=np.int32)\n    return encoded_tokens, encoded_masks","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_train_set(file_name: str, text_field: str, sentiment_fields: List[str],\n                   lang_field: str) -> Dict[str, List[Tuple[str, int]]]:\n    assert len(sentiment_fields) > 0, 'List of sentiment fields is empty!'\n    header = []\n    line_idx = 1\n    data_by_lang = dict()\n    with codecs.open(file_name, mode='r', encoding='utf-8', errors='ignore') as fp:\n        data_reader = csv.reader(fp, quotechar='\"', delimiter=',')\n        for row in data_reader:\n            if len(row) > 0:\n                err_msg = 'File \"{0}\": line {1} is wrong!'.format(file_name, line_idx)\n                if len(header) == 0:\n                    header = copy.copy(row)\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(text_field)\n                    assert text_field in header, err_msg2\n                    for cur_field in sentiment_fields:\n                        err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(\n                            cur_field)\n                        assert cur_field in header, err_msg2\n                    text_field_index = header.index(text_field)\n                    try:\n                        lang_field_index = header.index(lang_field)\n                    except:\n                        lang_field_index = -1\n                    indices_of_sentiment_fields = []\n                    for cur_field in sentiment_fields:\n                        indices_of_sentiment_fields.append(header.index(cur_field))\n                else:\n                    if len(row) == len(header):\n                        text = row[text_field_index].strip()\n                        assert len(text) > 0, err_msg + ' Text is empty!'\n                        if lang_field_index >= 0:\n                            cur_lang = row[lang_field_index].strip()\n                            assert len(cur_lang) > 0, err_msg + ' Language is empty!'\n                        else:\n                            cur_lang = 'en'\n                        max_proba = 0.0\n                        for cur_field_idx in indices_of_sentiment_fields:\n                            try:\n                                cur_proba = float(row[cur_field_idx])\n                            except:\n                                cur_proba = -1.0\n                            err_msg2 = err_msg + ' Value {0} is wrong!'.format(\n                                row[cur_field_idx]\n                            )\n                            assert (cur_proba >= 0.0) and (cur_proba <= 1.0), err_msg2\n                            if cur_proba > max_proba:\n                                max_proba = cur_proba\n                        new_label = 1 if max_proba >= 0.5 else 0\n                        if cur_lang not in data_by_lang:\n                            data_by_lang[cur_lang] = []\n                        data_by_lang[cur_lang].append((text, new_label))\n            if line_idx % 10000 == 0:\n                print('{0} lines of the \"{1}\" have been processed...'.format(line_idx,\n                                                                             file_name))\n            line_idx += 1\n    if line_idx > 0:\n        if (line_idx - 1) % 10000 != 0:\n            print('{0} lines of the \"{1}\" have been processed...'.format(line_idx - 1,\n                                                                         file_name))\n    return data_by_lang","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_test_set(file_name: str, id_field: str, text_field: str,\n                  lang_field: str) -> Dict[str, List[Tuple[str, int]]]:\n    header = []\n    line_idx = 1\n    data_by_lang = dict()\n    with codecs.open(file_name, mode='r', encoding='utf-8', errors='ignore') as fp:\n        data_reader = csv.reader(fp, quotechar='\"', delimiter=',')\n        for row in data_reader:\n            if len(row) > 0:\n                err_msg = 'File \"{0}\": line {1} is wrong!'.format(file_name, line_idx)\n                if len(header) == 0:\n                    header = copy.copy(row)\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(text_field)\n                    assert text_field in header, err_msg2\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(id_field)\n                    assert id_field in header, err_msg2\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(lang_field)\n                    assert lang_field in header, err_msg2\n                    id_field_index = header.index(id_field)\n                    text_field_index = header.index(text_field)\n                    lang_field_index = header.index(lang_field)\n                else:\n                    if len(row) == len(header):\n                        try:\n                            id_value = int(row[id_field_index])\n                        except:\n                            id_value = -1\n                        err_msg2 = err_msg + ' {0} is wrong ID!'.format(\n                            row[id_field_index])\n                        assert id_value >= 0, err_msg2\n                        text = row[text_field_index].strip()\n                        assert len(text) > 0, err_msg + ' Text is empty!'\n                        if lang_field_index >= 0:\n                            cur_lang = row[lang_field_index].strip()\n                            assert len(cur_lang) > 0, err_msg + ' Language is empty!'\n                        else:\n                            cur_lang = 'en'\n                        if cur_lang not in data_by_lang:\n                            data_by_lang[cur_lang] = []\n                        data_by_lang[cur_lang].append((text, id_value))\n            if line_idx % 10000 == 0:\n                print('{0} lines of the \"{1}\" have been processed...'.format(line_idx,\n                                                                             file_name))\n            line_idx += 1\n    if line_idx > 0:\n        if (line_idx - 1) % 10000 != 0:\n            print('{0} lines of the \"{1}\" have been processed...'.format(line_idx - 1,\n                                                                         file_name))\n    return data_by_lang","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_siamese_dataset(texts: Dict[str, List[Tuple[str, int]]],\n                          dataset_size: int, tokenizer: XLMRobertaTokenizer,\n                          maxlen: int, batch_size: int,\n                          shuffle: bool) -> Tuple[tf.data.Dataset, int]:\n    language_pairs = set()\n    for language in texts.keys():\n        for other_language in texts:\n            if other_language == language:\n                language_pairs.add((language, other_language))\n            else:\n                new_pair = (language, other_language)\n                new_pair_2 = (other_language, language)\n                if (new_pair not in language_pairs) and (new_pair_2 not in language_pairs):\n                    language_pairs.add(new_pair)\n    language_pairs = sorted(list(language_pairs))\n    print('Possible language pairs are: {0}.'.format(language_pairs))\n    err_msg = '{0} is too small size of the data set!'.format(dataset_size)\n    assert dataset_size >= (len(language_pairs) * 10), err_msg\n    n_samples_for_lang_pair = int(np.ceil(dataset_size / float(len(language_pairs))))\n    text_pairs_and_labels = []\n    for left_lang, right_lang in language_pairs:\n        print('{0}-{1}:'.format(left_lang, right_lang))\n        left_positive_indices = list(filter(\n            lambda idx: texts[left_lang][idx][1] > 0, range(len(texts[left_lang]))\n        ))\n        left_negative_indices = list(filter(\n            lambda idx: texts[left_lang][idx][1] == 0, range(len(texts[left_lang]))\n        ))\n        right_positive_indices = list(filter(\n            lambda idx: texts[right_lang][idx][1] > 0, range(len(texts[right_lang]))\n        ))\n        right_negative_indices = list(filter(\n            lambda idx: texts[right_lang][idx][1] == 0, range(len(texts[right_lang]))\n        ))\n        used_pairs = set()\n        number_of_samples = 0\n        for _ in range(n_samples_for_lang_pair // 4):\n            left_idx = random.choice(left_positive_indices)\n            right_idx = random.choice(right_positive_indices)\n            counter = 0\n            while ((right_idx == left_idx) or ((left_idx, right_idx) in used_pairs) or\n                   ((right_idx, left_idx) in used_pairs)) and (counter < 100):\n                right_idx = random.choice(right_positive_indices)\n                counter += 1\n            if counter < 100:\n                used_pairs.add((left_idx, right_idx))\n                used_pairs.add((right_idx, left_idx))\n                text_pairs_and_labels.append(\n                    (\n                        texts[left_lang][left_idx][0],\n                        texts[right_lang][right_idx][0],\n                        1\n                    )\n                )\n                number_of_samples += 1\n        print('  number of \"1-1\" pairs is {0};'.format(number_of_samples))\n        number_of_samples = 0\n        for _ in range(n_samples_for_lang_pair // 4, (2 * n_samples_for_lang_pair) // 4):\n            left_idx = random.choice(left_negative_indices)\n            right_idx = random.choice(right_negative_indices)\n            counter = 0\n            while ((right_idx == left_idx) or ((left_idx, right_idx) in used_pairs) or\n                   ((right_idx, left_idx) in used_pairs)) and (counter < 100):\n                right_idx = random.choice(right_negative_indices)\n                counter += 1\n            if counter < 100:\n                used_pairs.add((left_idx, right_idx))\n                used_pairs.add((right_idx, left_idx))\n                text_pairs_and_labels.append(\n                    (\n                        texts[left_lang][left_idx][0],\n                        texts[right_lang][right_idx][0],\n                        1\n                    )\n                )\n                number_of_samples += 1\n        print('  number of \"0-0\" pairs is {0};'.format(number_of_samples))\n        number_of_samples = 0\n        for _ in range((2 * n_samples_for_lang_pair) // 4, n_samples_for_lang_pair):\n            left_idx = random.choice(left_negative_indices)\n            right_idx = random.choice(right_positive_indices)\n            counter = 0\n            while ((right_idx == left_idx) or ((left_idx, right_idx) in used_pairs) or\n                   ((right_idx, left_idx) in used_pairs)) and (counter < 100):\n                right_idx = random.choice(right_positive_indices)\n                counter += 1\n            if counter < 100:\n                used_pairs.add((left_idx, right_idx))\n                used_pairs.add((right_idx, left_idx))\n                if random.random() >= 0.5:\n                    text_pairs_and_labels.append(\n                        (\n                            texts[left_lang][left_idx][0],\n                            texts[right_lang][right_idx][0],\n                            0\n                        )\n                    )\n                else:\n                    text_pairs_and_labels.append(\n                        (\n                            texts[right_lang][right_idx][0],\n                            texts[left_lang][left_idx][0],\n                            0\n                        )\n                    )\n                number_of_samples += 1\n        print('  number of \"0-1\" or \"1-0\" pairs is {0}.'.format(number_of_samples))\n    random.shuffle(text_pairs_and_labels)\n    n_steps = len(text_pairs_and_labels) // batch_size\n    print('Samples number of the data set is {0}.'.format(len(text_pairs_and_labels)))\n    print('Samples number per each language pair is {0}.'.format(n_samples_for_lang_pair))\n    tokens_of_left_texts, mask_of_left_texts = regular_encode(\n        texts=[cur[0] for cur in text_pairs_and_labels],\n        tokenizer=tokenizer, maxlen=maxlen\n    )\n    tokens_of_right_texts, mask_of_right_texts = regular_encode(\n        texts=[cur[1] for cur in text_pairs_and_labels],\n        tokenizer=tokenizer, maxlen=maxlen\n    )\n    siamese_labels = np.array([cur[2] for cur in text_pairs_and_labels], dtype=np.int32)\n    print('Number of positive siamese samples is {0} from {1}.'.format(\n        int(sum(siamese_labels)), siamese_labels.shape[0]))\n    if shuffle:\n        err_msg = '{0} is too small number of samples for the data set!'.format(\n            len(text_pairs_and_labels))\n        assert n_steps >= 50, err_msg\n        dataset = tf.data.Dataset.from_tensor_slices(\n            (\n                (\n                    tokens_of_left_texts, mask_of_left_texts,\n                    tokens_of_right_texts, mask_of_right_texts\n                ),\n                siamese_labels\n            )\n        ).repeat().batch(batch_size)\n    else:\n        dataset = tf.data.Dataset.from_tensor_slices(\n            (\n                (\n                    tokens_of_left_texts, mask_of_left_texts,\n                    tokens_of_right_texts, mask_of_right_texts\n                ),\n                siamese_labels\n            )\n        ).batch(batch_size)\n    del text_pairs_and_labels\n    return dataset, n_steps","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_feature_extractor(transformer_name: str, hidden_state_size: int,\n                            max_len: int) -> tf.keras.Model:\n    word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                     name=\"base_word_ids\")\n    attention_mask = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                           name=\"base_attention_mask\")\n    transformer_layer = TFXLMRobertaModel.from_pretrained(\n        pretrained_model_name_or_path=transformer_name,\n        name='Transformer'\n    )\n    sequence_output = transformer_layer([word_ids, attention_mask])[0]\n    output_mask = MaskCalculator(\n        output_dim=hidden_state_size, trainable=False,\n        name='OutMaskCalculator'\n    )(attention_mask)\n    masked_sequence_output = tf.keras.layers.Multiply(\n        name='OutMaskMultiplicator'\n    )([output_mask, sequence_output])\n    masked_sequence_output = tf.keras.layers.Masking(\n        name='OutMasking'\n    )(masked_sequence_output)\n    pooled_output = tf.keras.layers.GlobalAvgPool1D(name='AvePool')(masked_sequence_output)\n    text_embedding = tf.keras.layers.Lambda(\n        lambda x: tf.math.l2_normalize(x, axis=1),\n        name='Emdedding'\n    )(pooled_output)\n    fe_model = tf.keras.Model(\n        inputs=[word_ids, attention_mask],\n        outputs=text_embedding,\n        name='FeatureExtractor'\n    )\n    fe_model.build(input_shape=[(None, max_len), (None, max_len)])\n    return fe_model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def euclidean_distance(vects):\n    x, y = vects\n    sum_square = tf.keras.backend.sum(tf.keras.backend.square(x - y),\n                                      axis=1, keepdims=True)\n    return tf.keras.backend.sqrt(\n        tf.keras.backend.maximum(sum_square, tf.keras.backend.epsilon())\n    )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def eucl_dist_output_shape(shapes):\n    shape1, shape2 = shapes\n    return (shape1[0], 1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_siamese_nn(transformer_name: str, hidden_state_size: int, max_len: int,\n                     lr: float) -> Tuple[tf.keras.Model, tf.keras.Model]:\n    left_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                          name=\"left_word_ids\")\n    left_attention_mask = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                                name=\"left_attention_mask\")\n    right_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                           name=\"right_word_ids\")\n    right_attention_mask = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                                 name=\"right_attention_mask\")\n    fe_ = build_feature_extractor(transformer_name, hidden_state_size, max_len)\n    left_text_embedding = fe_([left_word_ids, left_attention_mask])\n    right_text_embedding = fe_([right_word_ids, right_attention_mask])\n    distance_layer = tf.keras.layers.Lambda(\n        function=euclidean_distance,\n        output_shape=eucl_dist_output_shape,\n        name='L2DistLayer'\n    )([left_text_embedding, right_text_embedding])\n    nn = tf.keras.Model(\n        inputs=[left_word_ids, left_attention_mask, right_word_ids, right_attention_mask],\n        outputs=distance_layer,\n        name='SiameseXLMR'\n    )\n    nn.compile(\n        optimizer=tfa.optimizers.AdamW(learning_rate=lr, weight_decay=1e-5),\n        loss=DBLLogLoss()\n    )\n    fe_.summary()\n    nn.summary()\n    return nn, fe_","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_classifier(language: str, feature_vector_size: int, n_train_samples: int,\n                     hidden_layer_ratio: float, n_hidden_layers: int,\n                     C: float, verbose: bool) -> tf.keras.Model:\n    sentence_features = tf.keras.layers.Input(\n        shape=(feature_vector_size,), dtype=tf.float32,\n        name=\"SentenceFeatures_{0}\".format(language)\n    )\n    n_samples = tf.cast(float(n_train_samples), dtype=tf.float32)\n    kl_weight = tf.cast(1.0 / C, dtype=tf.float32)\n    kl_divergence_function = (\n        lambda q, p, _: (tfp.distributions.kl_divergence(q, p) * kl_weight / n_samples)\n    )\n    if n_hidden_layers > 0:\n        hidden_layer_size = hidden_layer_ratio * 1.5 * feature_vector_size\n        hidden_layer = tfp.layers.DenseFlipout(\n            units=max(int(round(hidden_layer_size)), 10),\n            kernel_divergence_fn=kl_divergence_function,\n            bias_divergence_fn=kl_divergence_function,\n            activation=None, seed=generate_random_seed(),\n            name='BayesianHiddenLayer1_{0}'.format(language)\n        )(sentence_features)\n        hidden_layer = tf.keras.layers.LayerNormalization(\n            name='LayerNorm1_{0}'.format(language)\n        )(hidden_layer)\n        hidden_layer = tf.keras.layers.Activation(\n            activation=tf.keras.activations.relu,\n            name='Activation1_{0}'.format(language)\n        )(hidden_layer)\n        for layer_idx in range(1, n_hidden_layers):\n            hidden_layer_size *= hidden_layer_ratio\n            hidden_layer = tfp.layers.DenseFlipout(\n                units=max(int(round(hidden_layer_size)), 10),\n                kernel_divergence_fn=kl_divergence_function,\n                bias_divergence_fn=kl_divergence_function,\n                activation=None, seed=generate_random_seed(),\n                name='BayesianHiddenLayer{0}_{1}'.format(layer_idx + 1, language)\n            )(hidden_layer)\n            hidden_layer = tf.keras.layers.LayerNormalization(\n                name='LayerNorm{0}_{1}'.format(layer_idx + 1, language)\n            )(hidden_layer)\n            hidden_layer = tf.keras.layers.Activation(\n                activation=tf.keras.activations.relu,\n                name='Activation{0}_{1}'.format(layer_idx + 1, language)\n            )(hidden_layer)\n        cls_layer = tfp.layers.DenseFlipout(\n            units=1,\n            kernel_divergence_fn=kl_divergence_function,\n            bias_divergence_fn=kl_divergence_function,\n            activation='sigmoid',\n            name='BayesianOutputLayer_{0}'.format(language),\n            seed=generate_random_seed()\n        )(hidden_layer)\n    else:\n        cls_layer = tfp.layers.DenseFlipout(\n            units=1,\n            kernel_divergence_fn=kl_divergence_function,\n            bias_divergence_fn=kl_divergence_function,\n            activation='sigmoid',\n            name='BayesianOutputLayer_{0}'.format(language),\n            seed=generate_random_seed()\n        )(sentence_features)\n    cls_model = tf.keras.Model(\n        inputs=sentence_features,\n        outputs=cls_layer,\n        name='BayesianNN_{0}'.format(language)\n    )\n    radam = tfa.optimizers.RectifiedAdam(learning_rate=1e-2)\n    ranger = tfa.optimizers.Lookahead(radam, sync_period=6, slow_step_size=0.5)\n    cls_model.compile(\n        optimizer=ranger, loss='binary_crossentropy',\n        experimental_run_tf_function=False\n    )\n    if verbose:\n        cls_model.summary()\n    return cls_model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_datasets_for_classifier(\n        data_for_training: Tuple[np.ndarray, np.ndarray],\n        data_split: Dict[str, Tuple[np.ndarray, np.ndarray]],\n        language_for_testing: str\n) -> Tuple[Tuple[np.ndarray, np.ndarray], Tuple[np.ndarray, np.ndarray]]:\n    indices_for_training, indices_for_testing = data_split[language_for_testing]\n    X_train = data_for_training[0][indices_for_training]\n    y_train = data_for_training[1][indices_for_training]\n    X_test = data_for_training[0][indices_for_testing]\n    y_test = data_for_training[1][indices_for_testing]\n    del indices_for_training, indices_for_testing\n    data_for_training = (X_train, y_train)\n    data_for_testing = (X_test, y_test)\n    return data_for_training, data_for_testing","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_classifier(trainset: tf.data.Dataset, n_steps: int, bayesian_classifier: tf.keras.Model,\n                     max_epochs: int, verbose: bool, tmp_file_name: str):\n    callbacks = [\n        tf.keras.callbacks.ReduceLROnPlateau(\n            monitor='loss', factor=0.1, patience=3, verbose=False,\n            mode='min', min_delta=0.05, cooldown=5, min_lr=1e-7\n        ),\n        tf.keras.callbacks.ModelCheckpoint(\n            monitor='loss', mode=\"min\", save_weights_only=True, save_best_only=True,\n            filepath=tmp_file_name\n        )\n    ]\n    if verbose:\n        weight_posterior_callback = WeightPosteriorCallback(\n            log_every = max(2, max_epochs // 10)\n        )\n        callbacks.append(weight_posterior_callback)\n    else:\n        weight_posterior_callback = None\n    if verbose:\n        print('n_epochs = {0}, steps_per_epoch = {1}'.format(max_epochs, n_steps))\n    history = bayesian_classifier.fit(\n        trainset,\n        steps_per_epoch=n_steps,\n        epochs=max_epochs, callbacks=callbacks,\n        verbose=verbose\n    )\n    if verbose:\n        show_training_process(history, 'loss', figure_id=1)\n        plot_weight_posteriors(layer_names=weight_posterior_callback.layer_names,\n                               logged_epochs=weight_posterior_callback.logged_epochs)\n    del history, callbacks\n    if os.path.exists(tmp_file_name):\n        bayesian_classifier.load_weights(tmp_file_name)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evaluate_classifier(X_test: np.ndarray, y_test: np.ndarray, language_for_testing: str,\n                        bayesian_nn: tf.keras.Model,\n                        n_monte_carlo: int, batch_size: int, verbose: bool) -> float:\n    probabilities = predict_with_model(\n        classifier=bayesian_nn,\n        input_data=X_test,\n        batch_size=batch_size,\n        n_monte_carlo=n_monte_carlo\n    )\n    err_msg = '{0} != {1}'.format(y_test.shape, probabilities.shape)\n    assert y_test.shape == probabilities.shape, err_msg\n    if verbose:\n        show_roc_auc(y_true=y_val, probabilities=probabilities,\n                     label='the testing data (language \"{0}\")'.format(language_for_testing),\n                     figure_id=3)\n    quality = average_precision_score(y_true=y_test, y_score=probabilities)\n    del probabilities\n    return quality","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_training_process(history: tf.keras.callbacks.History, metric_name: str,\n                          figure_id: int=1):\n    val_metric_name = 'val_' + metric_name\n    err_msg = 'The metric \"{0}\" is not found! Available metrics are: {1}'.format(\n        metric_name, list(history.history.keys()))\n    assert metric_name in history.history, err_msg\n    plt.figure(figure_id, figsize=(5, 5))\n    plt.plot(list(range(len(history.history[metric_name]))),\n             history.history[metric_name], label='Training {0}'.format(metric_name))\n    if val_metric_name in history.history:\n        assert len(history.history[metric_name]) == len(history.history['val_' + metric_name])\n        plt.plot(list(range(len(history.history['val_' + metric_name]))),\n                 history.history['val_' + metric_name], label='Validation {0}'.format(metric_name))\n    plt.xlabel('Epochs')\n    plt.ylabel(metric_name)\n    plt.title('Training process')\n    plt.legend(loc='best')\n    plt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_siamese_nn(nn: tf.keras.Model, trainset: tf.data.Dataset, steps_per_trainset: int,\n                     steps_per_epoch: int, validset: tf.data.Dataset, max_duration: int,\n                     siamese_file_name: str):\n    assert steps_per_trainset >= steps_per_epoch\n    n_epochs = int(round(10.0 * steps_per_trainset / float(steps_per_epoch)))\n    print('Maximal duration of the Siamese NN training is {0} seconds.'.format(max_duration))\n    callbacks = [\n        tf.keras.callbacks.EarlyStopping(patience=5, monitor='val_loss', mode='min',\n                                         restore_best_weights=False, verbose=True),\n        tf.keras.callbacks.ModelCheckpoint(monitor='val_loss', mode=\"min\",\n                                           save_weights_only=True, save_best_only=True,\n                                           filepath=siamese_file_name),\n        tfa.callbacks.TimeStopping(seconds=max_duration, verbose=True)\n    ]\n    history = nn.fit(\n        trainset,\n        steps_per_epoch=steps_per_epoch,\n        validation_data=validset,\n        epochs=n_epochs,\n        callbacks=callbacks\n    )\n    show_training_process(history, 'loss')\n    nn.load_weights(siamese_file_name)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_roc_auc(y_true: np.ndarray, probabilities: np.ndarray, label: str,\n                 figure_id: int=1):\n    plt.figure(figure_id, figsize=(5, 5))\n    plt.plot([0, 1], [0, 1], 'k--')\n    print('ROC-AUC score for {0} is {1:.9f}'.format(\n        label, roc_auc_score(y_true=y_true, y_score=probabilities)\n    ))\n    fpr, tpr, _ = roc_curve(y_true=y_true, y_score=probabilities)\n    plt.plot(fpr, tpr, label=label.title())\n    plt.xlabel('False positive rate')\n    plt.ylabel('True positive rate')\n    plt.title('ROC curve')\n    plt.legend(loc='best')\n    plt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def calculate_features_of_texts(texts: Dict[str, List[Tuple[str, int]]],\n                                tokenizer: XLMRobertaTokenizer, maxlen: int,\n                                fe: tf.keras.Model, batch_size: int,\n                                max_dataset_size: int = 0) -> \\\n        Dict[str, Tuple[np.ndarray, np.ndarray]]:\n    languages = sorted(list(texts.keys()))\n    datasets_by_languages = dict()\n    if max_dataset_size > 0:\n        max_size_per_lang = max_dataset_size // len(languages)\n        err_msg = '{0} is too small number of dataset samples!'.format(max_dataset_size)\n        assert max_size_per_lang > 0, err_msg\n    else:\n        max_size_per_lang = 0\n    for cur_lang in languages:\n        selected_indices = list(range(len(texts[cur_lang])))\n        if max_size_per_lang > 0:\n            if len(selected_indices) > max_size_per_lang:\n                selected_indices = random.sample(\n                    population=selected_indices,\n                    k=max_size_per_lang\n                )\n        tokens_of_texts, mask_of_texts = regular_encode(\n            texts=[texts[cur_lang][idx][0] for idx in selected_indices],\n            tokenizer=tokenizer, maxlen=maxlen\n        )\n        X = []\n        n_batches = int(np.ceil(len(selected_indices) / float(batch_size)))\n        for batch_idx in range(n_batches):\n            batch_start = batch_idx * batch_size\n            batch_end = min(len(selected_indices), batch_start + batch_size)\n            res = fe.predict_on_batch(\n                [\n                    tokens_of_texts[batch_start:batch_end],\n                    mask_of_texts[batch_start:batch_end]\n                ]\n            )\n            if not isinstance(res, np.ndarray):\n                res = res.numpy()\n            X.append(res)\n            del res\n        X = np.vstack(X)\n        y = np.array([texts[cur_lang][idx][1] for idx in selected_indices], dtype=np.int32)\n        datasets_by_languages[cur_lang] = (X, y)\n        del X, y, selected_indices\n    return datasets_by_languages","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def predict_with_model(classifier: tf.keras.Model, input_data: np.ndarray,\n                       batch_size: int, n_monte_carlo: int) -> np.ndarray:\n    assert n_monte_carlo > 1\n    predicted = classifier.predict(input_data, batch_size=batch_size).flatten()\n    for _ in range(n_monte_carlo - 1):\n        predicted += classifier.predict(input_data, batch_size=batch_size).flatten()\n    return predicted / float(n_monte_carlo)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def select_best_model(data_for_training: List[Dict[str, Tuple[np.ndarray, np.ndarray]]],\n                      languages_for_training: List[str],\n                      current_strategy: tf.distribute.Strategy,\n                      tpu_system: Union[None, tf.distribute.cluster_resolver.TPUClusterResolver],\n                      feature_vector_size: int, hidden_layer_ratio: float,\n                      n_monte_carlo: int, max_iters: int, batch_size: int,\n                      tmp_file_name: str) -> Dict[str, float]:\n    assert len(data_for_training) == len(languages_for_training)\n    space  = [Real(1e-1, 1e+4, \"log-uniform\", name='C'),\n              Integer(0, 6, name='neural_depth')]\n    n_calls = 72\n    n_random_starts = 18\n    n_restarts_optimizer = 6\n    restart_counter = 1\n    datasets_for_training = []\n    for lang_idx in range(len(languages_for_training)):\n        n_train_samples = data_for_training[lang_idx]['train'][1].shape[0]\n        steps_per_epoch = int(np.ceil(n_train_samples / float(batch_size)))\n        datasets_for_training.append(\n            {\n                'train': tf.data.Dataset.from_tensor_slices(\n                    data_for_training[lang_idx]['train']\n                ).repeat().shuffle(n_train_samples).batch(batch_size),\n                'n_epochs': int(np.ceil(max_iters / float(n_train_samples))),\n                'steps_per_epoch': steps_per_epoch\n            }\n        )\n    \n    @use_named_args(space)\n    def objective_f(C: float, neural_depth: int) -> float:\n        nonlocal restart_counter\n        nonlocal current_strategy\n        nonlocal tpu_system\n        pr_scores = []\n        n_splits = len(languages_for_training)\n        for lang_idx in range(len(languages_for_training)):\n            validation_lang = languages_for_training[lang_idx]\n            with current_strategy.scope():\n                bnn = build_classifier(\n                    feature_vector_size=feature_vector_size, n_train_samples=max_iters,\n                    hidden_layer_ratio=hidden_layer_ratio, n_hidden_layers=neural_depth,\n                    C=float(C), verbose=False,\n                    language='{0}{1}'.format(validation_lang.strip(), restart_counter)\n                )\n            train_classifier(\n                trainset=datasets_for_training[lang_idx]['train'],\n                n_steps=datasets_for_training[lang_idx]['steps_per_epoch'],\n                bayesian_classifier=bnn, verbose=False,\n                max_epochs=datasets_for_training[lang_idx]['n_epochs'],\n                tmp_file_name=tmp_file_name\n            )\n            instant_quality = evaluate_classifier(\n                X_test=data_for_training[lang_idx]['test'][0],\n                y_test=data_for_training[lang_idx]['test'][1],\n                language_for_testing=languages_for_training[lang_idx],\n                bayesian_nn=bnn,\n                n_monte_carlo=n_monte_carlo, batch_size=batch_size, verbose=False\n            )\n            pr_scores.append(instant_quality)\n            del bnn\n            gc.collect()\n            tf.keras.backend.clear_session()\n            if os.path.isfile(tmp_file_name):\n                os.remove(tmp_file_name)\n        pr_score = hmean(pr_scores)\n        del pr_scores\n        print('  C={0:.9f}, neural_depth={1}'.format(C, neural_depth))\n        print('  Precision-Recall score = {0:.9f}'.format(pr_score))\n        restart_counter += 1\n        if tpu_system:\n            tf.tpu.experimental.shutdown_tpu_system(tpu_system)\n            del current_strategy\n            tf.tpu.experimental.initialize_tpu_system(tpu_system)\n            current_strategy = tf.distribute.experimental.TPUStrategy(tpu_system)\n        return -pr_score\n    \n    start_time = time.time()\n    res_gp = gp_minimize(\n        objective_f, space,\n        n_calls=n_calls, n_random_starts=n_random_starts,\n        n_restarts_optimizer=n_restarts_optimizer, random_state=42,\n        verbose=True, n_jobs=1\n    )\n    best_parameters = {\n        'C': float(res_gp.x[0]),\n        'neural_depth': int(res_gp.x[1]),\n    }\n    automl_duration = int(round(time.time() - start_time))\n    print('')\n    print('Total duration of the AutoML is {0} seconds.'.format(automl_duration))\n    print('')\n    print('Best parameters are:')\n    print('C={0:.9f}, neural_depth={1}'.format(best_parameters['C'],\n                                               best_parameters['neural_depth']))\n    print('')\n    del datasets_for_training\n    plot_convergence(res_gp)\n    plot_evaluations(res_gp, bins=10)\n    return best_parameters","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"experiment_start_time = time.time()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n    model_name = 'jplu/tf-xlm-roberta-large'\n    max_seq_len = 256\n    batch_size_for_siamese = 8 * strategy.num_replicas_in_sync\nelse:\n    strategy = tf.distribute.get_strategy()\n    physical_devices = tf.config.list_physical_devices('GPU')\n    for device_idx in range(strategy.num_replicas_in_sync):\n        tf.config.experimental.set_memory_growth(physical_devices[device_idx], True)\n    max_seq_len = 256\n    model_name = 'jplu/tf-xlm-roberta-base'\n    batch_size_for_siamese = 4 * strategy.num_replicas_in_sync\nbatch_size_for_cls = max(8, 64 // strategy.num_replicas_in_sync) * strategy.num_replicas_in_sync\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\nprint('Model name: {0}'.format(model_name))\nprint('Maximal length of sequence is {0}'.format(max_seq_len))\nprint('Batch size for the Siamese XLM-RoBERTa is {0}'.format(\n    batch_size_for_siamese))\nprint('Batch size for the Bayesian NN is {0}'.format(\n    batch_size_for_cls))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random.seed(42)\nnp.random.seed(42)\ntf.random.set_seed(42)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tfa.register_all()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"siamese_learning_rate = 1e-5\nautoml_num_monte_carlo = 50\nfinal_num_monte_carlo = 100\ncls_layer_scale_coeff = 0.7\nmax_iters_of_cls = 60000\ndataset_dir = '/kaggle/input/jigsaw-multilingual-toxic-comment-classification'\ntmp_roberta_name = '/kaggle/working/siamese_xlmr.h5'\nfeature_extractor_dir = '/kaggle/working'\ntmp_cls_name = '/kaggle/working/bayesian_cls.h5'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xlmroberta_tokenizer = AutoTokenizer.from_pretrained(model_name)\nxlmroberta_config = XLMRobertaConfig.from_pretrained(model_name)\nprint(xlmroberta_config)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sentence_embedding_size = xlmroberta_config.hidden_size\nprint('Sentence embedding size is {0}'.format(sentence_embedding_size))\nassert max_seq_len <= xlmroberta_config.max_position_embeddings\nhidden_layer_ratio = 0.7","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corpus_for_training = load_train_set(\n    os.path.join(dataset_dir, \"jigsaw-toxic-comment-train.csv\"),\n    text_field=\"comment_text\", lang_field=\"lang\",\n    sentiment_fields=[\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\",\n                      \"identity_hate\"]\n)\nassert 'en' in corpus_for_training","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random.shuffle(corpus_for_training['en'])\nn_validation = int(round(0.15 * len(corpus_for_training['en'])))\ncorpus_for_validation = {'en': corpus_for_training['en'][:n_validation]}\ncorpus_for_training = {'en': corpus_for_training['en'][n_validation:]}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"multilingual_corpus = load_train_set(\n    os.path.join(dataset_dir, \"validation.csv\"),\n    text_field=\"comment_text\", lang_field=\"lang\", sentiment_fields=[\"toxic\", ]\n)\nassert 'en' not in multilingual_corpus\nmax_size = 0\nprint('Multilingual data:')\nfor language in sorted(list(multilingual_corpus.keys())):\n    print('  {0}\\t\\t{1} samples'.format(language, len(multilingual_corpus[language])))\n    assert set(map(lambda cur: cur[1], multilingual_corpus[language])) == {0, 1}\n    if len(multilingual_corpus[language]) > max_size:\n        max_size = len(multilingual_corpus[language])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"texts_for_submission = load_test_set(\n    os.path.join(dataset_dir, \"test.csv\"),\n    text_field=\"content\", lang_field=\"lang\", id_field=\"id\"\n)\nprint('Data for submission:')\nfor language in sorted(list(texts_for_submission.keys())):\n    print('  {0}\\t\\t{1} samples'.format(language, len(texts_for_submission[language])))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_for_training, n_batches_per_data = build_siamese_dataset(\n    texts=corpus_for_training, dataset_size=150000,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    batch_size=batch_size_for_siamese, shuffle=True\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_for_validation, n_batches_per_epoch = build_siamese_dataset(\n    texts=corpus_for_validation, dataset_size=1000,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    batch_size=batch_size_for_siamese, shuffle=False\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del corpus_for_training, corpus_for_validation\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preparing_duration = int(round(time.time() - experiment_start_time))\nprint(\"Duration of data loading and preparing to the Siamese NN training is \"\n      \"{0} seconds.\".format(preparing_duration))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    siamese_network, feature_extractor = build_siamese_nn(\n        transformer_name=model_name,\n        hidden_state_size=sentence_embedding_size,\n        max_len=max_seq_len,\n        lr=siamese_learning_rate\n    )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_siamese_nn(nn=siamese_network, trainset=dataset_for_training,\n                 steps_per_trainset=n_batches_per_data,\n                 steps_per_epoch=min(5 * n_batches_per_epoch, n_batches_per_data),\n                 validset=dataset_for_validation,\n                 max_duration=int(round(1.5 * 3600.0 - preparing_duration)),\n                 siamese_file_name=tmp_roberta_name)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del dataset_for_training\ndel dataset_for_validation\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_extractor.get_layer('Transformer' ).save_pretrained(feature_extractor_dir)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if os.path.isfile(tmp_roberta_name):\n    os.remove(tmp_roberta_name)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del siamese_network\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_for_training = calculate_features_of_texts(\n    texts=multilingual_corpus,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    fe=feature_extractor,\n    batch_size=batch_size_for_siamese\n)\nassert len(dataset_for_training) == 3","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_for_submission = calculate_features_of_texts(\n    texts=texts_for_submission,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    fe=feature_extractor,\n    batch_size=batch_size_for_siamese\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_embedded = []\ny_embedded = []\nsplit_by_languages = dict()\nstart_pos = 0\nfor cur_lang in dataset_for_training:\n    X_embedded.append(dataset_for_training[cur_lang][0])\n    y_embedded.append(dataset_for_training[cur_lang][1])\n    split_by_languages[cur_lang] = (\n        set(),\n        set(range(start_pos, start_pos + dataset_for_training[cur_lang][1].shape[0]))\n    )\n    start_pos = start_pos + dataset_for_training[cur_lang][1].shape[0]\nfeatured_data_for_training = (\n    np.vstack(X_embedded),\n    np.concatenate(y_embedded)\n)\nfor cur_lang in dataset_for_training:\n    indices_for_testing = split_by_languages[cur_lang][1]\n    indices_for_training = set(range(featured_data_for_training[0].shape[0])) - indices_for_testing\n    split_by_languages[cur_lang] = (\n        np.array(sorted(list(indices_for_training)), dtype=np.int32),\n        np.array(sorted(list(indices_for_testing)), dtype=np.int32)\n    )\n    del indices_for_training, indices_for_testing\nfeatured_data_for_submission = []\nidentifies_for_submission = []\nfor cur_lang in dataset_for_submission:\n    X_embedded.append(dataset_for_submission[cur_lang][0])\n    featured_data_for_submission.append(dataset_for_submission[cur_lang][0])\n    identifies_for_submission.append(dataset_for_submission[cur_lang][1])\n    y_embedded.append(\n        np.array(\n            [-1 for _ in range(dataset_for_submission[cur_lang][0].shape[0])],\n            dtype=np.int32\n        )\n    )\nfeatured_data_for_submission = np.vstack(featured_data_for_submission)\nidentifies_for_submission = np.concatenate(identifies_for_submission)\nX_embedded = np.vstack(X_embedded)\ny_embedded = np.concatenate(y_embedded)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del dataset_for_training, dataset_for_submission\ndel feature_extractor, xlmroberta_tokenizer","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_languages = sorted(list(split_by_languages.keys()))\nprev_lang = all_languages[0]\nassert len(set(split_by_languages[prev_lang][1].tolist()) & \\\n           set(split_by_languages[prev_lang][0].tolist())) == 0\nfor cur_lang in all_languages[1:]:\n    assert len(set(split_by_languages[cur_lang][1].tolist()) & \\\n               set(split_by_languages[cur_lang][0].tolist())) == 0\n    assert len(set(split_by_languages[cur_lang][1].tolist()) & \\\n               set(split_by_languages[prev_lang][1].tolist())) == 0\n    prev_lang = cur_lang","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"indices_of_samples = random.sample(\n    list(range(featured_data_for_training[1].shape[0])),\n    k=1000\n)\nindices_of_samples += random.sample(\n    list(range(featured_data_for_training[1].shape[0], y_embedded.shape[0])),\n    k=1000\n)\nX_embedded = X_embedded[indices_of_samples]\ny_embedded = y_embedded[indices_of_samples]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_embedded = TSNE(n_components=2, n_jobs=-1, random_state=42).fit_transform(X_embedded)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"indices_of_unknown_classes = list(filter(\n    lambda sample_idx: y_embedded[sample_idx] < 0,\n    range(len(y_embedded))\n))\nxy = X_embedded[indices_of_unknown_classes]\nplt.figure(figsize=(10, 10))\nplt.plot(xy[:, 0], xy[:, 1], 'o', color='b', markersize=2,\n         label='Unlabeled data')\nindices_of_negative_classes = list(filter(\n    lambda sample_idx: y_embedded[sample_idx] == 0,\n    range(len(y_embedded))\n))\nxy = X_embedded[indices_of_negative_classes]\nplt.plot(xy[:, 0], xy[:, 1], 'o', color='g', markersize=4,\n         label='Normal texts')\nindices_of_positive_classes = list(filter(\n    lambda sample_idx: y_embedded[sample_idx] > 0,\n    range(len(y_embedded))\n))\nxy = X_embedded[indices_of_positive_classes]\nplt.plot(xy[:, 0], xy[:, 1], 'o', color='r', markersize=6,\n         label='Toxic texts')\nplt.title('Toxic and normal texts')\nplt.legend(loc='best')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del indices_of_negative_classes\ndel indices_of_positive_classes\ndel indices_of_unknown_classes\ndel indices_of_samples\ndel X_embedded, y_embedded","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()\ntf.keras.backend.clear_session()\nif tpu:\n    tf.tpu.experimental.shutdown_tpu_system(tpu)\n    del strategy\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"splitted_data_for_training = []\nfor cur_lang in all_languages:\n    datasets = build_datasets_for_classifier(\n        data_for_training=featured_data_for_training,\n        data_split=split_by_languages,\n        language_for_testing=cur_lang\n    )\n    splitted_data_for_training.append(\n        {\n            'train': datasets[0],\n            'test': datasets[1]\n        }\n    )\n    del datasets","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"experiment_duration = int(round(time.time() - experiment_start_time))\nprint('Duration of siamese XLM-RoBERTa preparing is {0} seconds.'.format(\n    experiment_duration))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bnn_params = select_best_model(\n    data_for_training=splitted_data_for_training,\n    languages_for_training=all_languages,\n    current_strategy=strategy, tpu_system=tpu,\n    feature_vector_size=featured_data_for_training[0].shape[1],\n    hidden_layer_ratio=cls_layer_scale_coeff,\n    batch_size=batch_size_for_cls, max_iters=max_iters_of_cls,\n    n_monte_carlo=automl_num_monte_carlo,\n    tmp_file_name=tmp_cls_name\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del splitted_data_for_training","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if os.path.isfile(tmp_cls_name):\n    os.remove(tmp_cls_name)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_total_train_samples = featured_data_for_training[1].shape[0]\nn_steps_per_epoch = int(np.ceil(n_total_train_samples / float(batch_size_for_cls)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    final_bayesian_classifier = build_classifier(\n        feature_vector_size=featured_data_for_training[0].shape[1],\n        n_train_samples=max_iters_of_cls,\n        hidden_layer_ratio=cls_layer_scale_coeff,\n        n_hidden_layers=bnn_params['neural_depth'],\n        C=bnn_params['C'], language='multilang', verbose=True\n    )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_classifier(\n    trainset=tf.data.Dataset.from_tensor_slices(\n        featured_data_for_training\n    ).repeat().shuffle(n_total_train_samples).batch(batch_size_for_cls),\n    n_steps=n_steps_per_epoch,\n    bayesian_classifier=final_bayesian_classifier, verbose=True,\n    max_epochs=int(np.ceil(max_iters_of_cls / float(n_total_train_samples))),\\\n    tmp_file_name=tmp_cls_name\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_of_submission = predict_with_model(\n    classifier=final_bayesian_classifier,\n    input_data=featured_data_for_submission,\n    batch_size=batch_size_for_cls,\n    n_monte_carlo=final_num_monte_carlo\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert identifies_for_submission.shape == result_of_submission.shape","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with codecs.open('submission.csv', mode='w', encoding='utf-8', errors='ignore') as fp:\n    fp.write('id,toxic\\n')\n    for sample_idx in range(identifies_for_submission.shape[0]):\n        id_val = identifies_for_submission[sample_idx]\n        proba_val = result_of_submission[sample_idx]\n        fp.write('{0},{1:.9f}\\n'.format(id_val, proba_val))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Experiment duration is {0:.3f}.'.format(time.time() - experiment_start_time))","metadata":{},"execution_count":null,"outputs":[]}]}