{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this notebook\n\nThis notebook is part of my experiments with a cross-lingual text classifier based on metric learning with deep [Transformer](https://arxiv.org/abs/1706.03762)-based networks for feature transformation and on final classification in new feature space with another approach (for example, Bayesian neural network). The [Jigsaw Multilingual Toxic Comment Classification challenge](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification) is selected as a data source for labeled cross-lingual texts. This task has two nuances:\n\n1. I want to build a semantic space that is independent of a concrete language.\n\n2. I have a large labeled text corpus (hundreds of thousands of labeled texts) for a single language only, but training datasets for other languages are very small or they are empty at all. And I have to classify texts just in such languages.\n\nA transfer learning helps us to account for the first of these nuances. I use a pre-trained [XLM-RoBERTa](https://arxiv.org/abs/1911.02116) model as an initial state for a [Siamese neural network](https://link.springer.com/protocol/10.1007/978-1-0716-0826-5_3) that trains to transform a common semantic space into its special analog, into which a distance between tonally opposed texts is larger and distance between texts with the same sentiments is smaller. For the Siamese NN learning, I apply a special loss function which is known as [Distance Based Logistic Loss (a DBL loss)](https://arxiv.org/abs/1608.00161). Such loss is better than usual cross-entropy loss, because it is contrastive-based, and any contrastive-based loss guarantees that the Siamese neural network after its training will calculate a compact space with required semantic properties. In comparison with a \"classical\" contrastive loss, which is popular for Siamese neural networks, the DBL loss is more effective owing to quicker convergence. It is known, that a Triplet Loss is also used for the Siamese neural network, but in my experiments with textual data, this loss didn't allow well separable semantic space, and so I didn't include it in this notebook."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import codecs\nimport copy\nimport csv\nimport gc\nimport os\nimport pickle\nimport random\nimport time\nfrom typing import Dict, List, Tuple, Union","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom nltk import wordpunct_tokenize\nimport numpy as np\nfrom sklearn.manifold import TSNE\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf\nimport tensorflow_addons as tfa\nfrom tensorflow.python.framework import ops, tensor_util\nfrom tensorflow.python.keras.utils import losses_utils, tf_utils\nfrom tensorflow.python.ops import math_ops\nfrom tensorflow.python.ops.losses import util as tf_losses_util\nfrom tqdm.notebook import tqdm\nfrom transformers import AutoTokenizer, XLMRobertaTokenizer\nfrom transformers import TFXLMRobertaModel, XLMRobertaConfig","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class LossFunctionWrapper(tf.keras.losses.Loss):\n    def __init__(self,\n                 fn,\n                 reduction=losses_utils.ReductionV2.AUTO,\n                 name=None,\n                 **kwargs):\n        super(LossFunctionWrapper, self).__init__(reduction=reduction,\n                                                  name=name)\n        self.fn = fn\n        self._fn_kwargs = kwargs\n\n    def call(self, y_true, y_pred):\n        if tensor_util.is_tensor(y_pred) and tensor_util.is_tensor(y_true):\n            y_pred, y_true = tf_losses_util.squeeze_or_expand_dimensions(\n                y_pred, y_true\n            )\n        return self.fn(y_true, y_pred, **self._fn_kwargs)\n\n    def get_config(self):\n        config = {}\n        for k, v in six.iteritems(self._fn_kwargs):\n            config[k] = tf.keras.backend.eval(v) \\\n                if tf_utils.is_tensor_or_variable(v) \\\n                else v\n        base_config = super(LossFunctionWrapper, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def distance_based_log_loss(y_true, y_pred):\n    y_pred = ops.convert_to_tensor(y_pred)\n    y_true = math_ops.cast(y_true, y_pred.dtype)\n    margin = 1.0\n    p = (1.0 + tf.math.exp(-margin)) / (1.0 + tf.math.exp(y_pred - margin))\n    return tf.keras.backend.binary_crossentropy(target=y_true, output=p)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class DBLLogLoss(LossFunctionWrapper):\n    def __init__(self, reduction=losses_utils.ReductionV2.AUTO,\n                 name='distance_based_log_loss'):\n        super(DBLLogLoss, self).__init__(distance_based_log_loss, name=name,\n                                         reduction=reduction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class AttentionMaskLayer(tf.keras.layers.Layer):\n    def __init__(self, pad_token_id: int, **kwargs):\n        self.pad_token_id = pad_token_id\n        super(AttentionMaskLayer, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        super(AttentionMaskLayer, self).build(input_shape)\n\n    def call(self, inputs, **kwargs):\n        return tf.keras.backend.cast(\n            x=tf.math.not_equal(\n                x=inputs,\n                y=self.pad_token_id\n            ),\n            dtype='int32'\n        )\n\n    def compute_output_shape(self, input_shape):\n        return input_shape\n\n    def get_config(self):\n        return {\"pad_token_id\": self.pad_token_id}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def tokenize_all(...)**\n\nThis function transforms input texts to token IDs for XLM-RoBERTa. It returns a 2-d numpy array with integer values."},{"metadata":{"trusted":true},"cell_type":"code","source":"def tokenize_all(texts: List[str], tokenizer: XLMRobertaTokenizer,\n                 maxlen: int) -> List[List[int]]:\n    if not isinstance(texts, list):\n        err_msg = '\"{0}\" is wrong type for the text list!'.format(type(texts))\n        raise ValueError(err_msg)\n    n_texts = len(texts)\n    all_tokenized_texts = []\n    for cur_text in tqdm(texts):\n        full_words = wordpunct_tokenize(cur_text)\n        sub_words = []\n        for cur_word in filter(lambda it2: len(it2) > 0,\n                               map(lambda it1: it1.strip(), full_words)):\n            bpe = tokenizer.tokenize(cur_word)\n            if tokenizer.unk_token in bpe:\n                sub_words.append(tokenizer.unk_token)\n            else:\n                sub_words += bpe\n        sub_words = [tokenizer.bos_token] + sub_words + \\\n                    [tokenizer.eos_token]\n        if len(sub_words) > maxlen:\n            sub_words = sub_words[:maxlen]\n        elif len(sub_words) < maxlen:\n            ndiff = maxlen - len(sub_words)\n            for _ in range(ndiff):\n                sub_words.append(tokenizer.pad_token)\n        all_tokenized_texts.append(\n            tokenizer.convert_tokens_to_ids(sub_words)\n        )\n        del sub_words, full_words\n    return all_tokenized_texts","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def load_train_set(...)**\n\nThis function loads multilingual labeled text corpus for training from the CSV file. If there is no information about language in this CSV file, then I think that all texts language is English.\n    \nIt returns a special dictionary with information about texts and their binary toxicity labels (integer values) by languages (language is a key, and a list of texts and labels is a value)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_train_set(file_name: str, text_field: str, sentiment_fields: List[str],\n                   lang_field: str) -> Dict[str, List[Tuple[str, int]]]:\n    assert len(sentiment_fields) > 0, 'List of sentiment fields is empty!'\n    header = []\n    line_idx = 1\n    data_by_lang = dict()\n    with codecs.open(file_name, mode='r', encoding='utf-8', errors='ignore') as fp:\n        data_reader = csv.reader(fp, quotechar='\"', delimiter=',')\n        for row in data_reader:\n            if len(row) > 0:\n                err_msg = 'File \"{0}\": line {1} is wrong!'.format(file_name, line_idx)\n                if len(header) == 0:\n                    header = copy.copy(row)\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(text_field)\n                    assert text_field in header, err_msg2\n                    for cur_field in sentiment_fields:\n                        err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(\n                            cur_field)\n                        assert cur_field in header, err_msg2\n                    text_field_index = header.index(text_field)\n                    try:\n                        lang_field_index = header.index(lang_field)\n                    except:\n                        lang_field_index = -1\n                    indices_of_sentiment_fields = []\n                    for cur_field in sentiment_fields:\n                        indices_of_sentiment_fields.append(header.index(cur_field))\n                else:\n                    if len(row) == len(header):\n                        text = row[text_field_index].strip()\n                        assert len(text) > 0, err_msg + ' Text is empty!'\n                        if lang_field_index >= 0:\n                            cur_lang = row[lang_field_index].strip()\n                            assert len(cur_lang) > 0, err_msg + ' Language is empty!'\n                        else:\n                            cur_lang = 'en'\n                        max_proba = 0.0\n                        for cur_field_idx in indices_of_sentiment_fields:\n                            try:\n                                cur_proba = float(row[cur_field_idx])\n                            except:\n                                cur_proba = -1.0\n                            err_msg2 = err_msg + ' Value {0} is wrong!'.format(\n                                row[cur_field_idx]\n                            )\n                            assert (cur_proba >= 0.0) and (cur_proba <= 1.0), err_msg2\n                            if cur_proba > max_proba:\n                                max_proba = cur_proba\n                        new_label = 1 if max_proba >= 0.5 else 0\n                        if cur_lang not in data_by_lang:\n                            data_by_lang[cur_lang] = []\n                        data_by_lang[cur_lang].append((text, new_label))\n            if line_idx % 10000 == 0:\n                print('{0} lines of the \"{1}\" have been processed...'.format(\n                    line_idx, file_name\n                ))\n            line_idx += 1\n    if line_idx > 0:\n        if (line_idx - 1) % 10000 != 0:\n            print('{0} lines of the \"{1}\" have been processed...'.format(\n                line_idx - 1, file_name\n            ))\n    return data_by_lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def load_test_set(...)**\n\nThis function loads multilingual unlabeled text corpus for submission from the CSV file. If there is no information about language in this CSV file, then I think that all texts language is English.\n    \nIt returns a special dictionary with information about texts and their integer identifiers by languages (language is a key, and a list of texts and identifiers is a value).\n\nThe function differs from the previous load_train_set only in the expected structure (field set) of the parsed CSV file, and these functions are the same by the structure of returned objects (but special identifiers for submission are used instead of toxicity labels)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_test_set(file_name: str, id_field: str, text_field: str,\n                  lang_field: str) -> Dict[str, List[Tuple[str, int]]]:\n    header = []\n    line_idx = 1\n    data_by_lang = dict()\n    with codecs.open(file_name, mode='r', encoding='utf-8', errors='ignore') as fp:\n        data_reader = csv.reader(fp, quotechar='\"', delimiter=',')\n        for row in data_reader:\n            if len(row) > 0:\n                err_msg = 'File \"{0}\": line {1} is wrong!'.format(file_name, line_idx)\n                if len(header) == 0:\n                    header = copy.copy(row)\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(text_field)\n                    assert text_field in header, err_msg2\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(id_field)\n                    assert id_field in header, err_msg2\n                    err_msg2 = err_msg + ' Field \"{0}\" is not found!'.format(lang_field)\n                    assert lang_field in header, err_msg2\n                    id_field_index = header.index(id_field)\n                    text_field_index = header.index(text_field)\n                    lang_field_index = header.index(lang_field)\n                else:\n                    if len(row) == len(header):\n                        try:\n                            id_value = int(row[id_field_index])\n                        except:\n                            id_value = -1\n                        err_msg2 = err_msg + ' {0} is wrong ID!'.format(\n                            row[id_field_index])\n                        assert id_value >= 0, err_msg2\n                        text = row[text_field_index].strip()\n                        assert len(text) > 0, err_msg + ' Text is empty!'\n                        if lang_field_index >= 0:\n                            cur_lang = row[lang_field_index].strip()\n                            assert len(cur_lang) > 0, err_msg + ' Language is empty!'\n                        else:\n                            cur_lang = 'en'\n                        if cur_lang not in data_by_lang:\n                            data_by_lang[cur_lang] = []\n                        data_by_lang[cur_lang].append((text, id_value))\n            if line_idx % 10000 == 0:\n                print('{0} lines of the \"{1}\" have been processed...'.format(\n                    line_idx, file_name\n                ))\n            line_idx += 1\n    if line_idx > 0:\n        if (line_idx - 1) % 10000 != 0:\n            print('{0} lines of the \"{1}\" have been processed...'.format(\n                line_idx - 1, file_name\n            ))\n    return data_by_lang","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def generate_text_probabilities(...)**\n\nThis function generates probabilities of text This function generates probabilities of each text random selection from list of all texts. Selection probability is the greater, the shorter the corresponded text."},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_text_probabilities(source_texts: List[Tuple[str, int]],\n                                interesting_indices: List[int]) -> np.ndarray:\n    lengths_of_texts = []\n    max_chars_number = 0\n    for idx in interesting_indices:\n        cur_chars_number = len(source_texts[idx][0])\n        lengths_of_texts.append(cur_chars_number)\n        if cur_chars_number > max_chars_number:\n            max_chars_number = cur_chars_number\n    assert max_chars_number > 100\n    probabilities = np.zeros((len(lengths_of_texts),), dtype=np.float64)\n    counter = 0\n    for idx, val in enumerate(lengths_of_texts):\n        if val > 10:\n            probabilities[idx] = max_chars_number * 10 - val\n        else:\n            counter += 1\n    assert counter == 0\n    probabilities /= np.sum(probabilities)\n    min_proba = 0.5 / float(probabilities.shape[0])\n    for idx in range(probabilities.shape[0]):\n        if probabilities[idx] > 0.0:\n            if probabilities[idx] < min_proba:\n                probabilities[idx] = min_proba\n    return probabilities / np.sum(probabilities)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def build_siamese_dataset(...)**\n\nThis function transforms a labeled text corpus in format, which is like to result of the *load_train_set* function, into a special Tensorflow dataset with considering of specified mini-batch size. Also, it returns the total number of mini-batches in this dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_siamese_dataset(texts: Dict[str, List[Tuple[str, int]]],\n                          dataset_size: int, tokenizer: XLMRobertaTokenizer,\n                          maxlen: int, batch_size: int,\n                          shuffle: bool) -> Tuple[tf.data.Dataset, int]:\n    language_pairs = set()\n    for language in texts.keys():\n        for other_language in texts:\n            if other_language == language:\n                language_pairs.add((language, other_language))\n            else:\n                pair_1 = (language, other_language)\n                pair_2 = (other_language, language)\n                if (pair_1 not in language_pairs) and (pair_2 not in language_pairs):\n                    language_pairs.add(pair_1)\n    language_pairs = sorted(list(language_pairs))\n    print('Possible language pairs are: {0}.'.format(language_pairs))\n    err_msg = '{0} is too small size of the data set!'.format(dataset_size)\n    assert dataset_size >= (len(language_pairs) * 10), err_msg\n    n_samples_for_lang_pair = int(np.ceil(dataset_size / float(len(language_pairs))))\n    text_pairs_and_labels = []\n    for left_lang, right_lang in language_pairs:\n        print('{0}-{1}:'.format(left_lang, right_lang))\n        left_positive_indices = list(filter(\n            lambda idx: ((texts[left_lang][idx][1] > 0) and \\\n                         (len(texts[left_lang][idx][0]) > 10)),\n            range(len(texts[left_lang]))\n        ))\n        left_positive_probas = generate_text_probabilities(\n            source_texts=texts[left_lang],\n            interesting_indices=left_positive_indices\n        )\n        left_negative_indices = list(filter(\n            lambda idx: ((texts[left_lang][idx][1] == 0) and \\\n                         (len(texts[left_lang][idx][0]) > 10)),\n            range(len(texts[left_lang]))\n        ))\n        left_negative_probas = generate_text_probabilities(\n            source_texts=texts[left_lang],\n            interesting_indices=left_negative_indices\n        )\n        right_positive_indices = list(filter(\n            lambda idx: ((texts[right_lang][idx][1] > 0) and \\\n                         (len(texts[right_lang][idx][0]) > 10)),\n            range(len(texts[right_lang]))\n        ))\n        right_positive_probas = generate_text_probabilities(\n            source_texts=texts[right_lang],\n            interesting_indices=right_positive_indices\n        )\n        right_negative_indices = list(filter(\n            lambda idx: ((texts[right_lang][idx][1] == 0) and \\\n                         (len(texts[right_lang][idx][0]) > 10)),\n            range(len(texts[right_lang]))\n        ))\n        right_negative_probas = generate_text_probabilities(\n            source_texts=texts[right_lang],\n            interesting_indices=right_negative_indices\n        )\n        used_pairs = set()\n        number_of_samples = 0\n        iterations = n_samples_for_lang_pair // 4\n        if len(left_positive_indices) > iterations:\n            left_indices = np.random.choice(\n                left_positive_indices,\n                min(iterations * 2, len(left_positive_indices)),\n                p=left_positive_probas, replace=False\n            ).tolist()\n        else:\n            left_indices = left_positive_indices\n        if len(right_positive_indices) > iterations:\n            right_indices = np.random.choice(\n                right_positive_indices,\n                min(iterations * 2, len(right_positive_indices)),\n                p=right_positive_probas, replace=False\n            ).tolist()\n        else:\n            right_indices = right_positive_indices\n        if len(left_indices) < len(right_indices):\n            right_indices = right_indices[:len(left_indices)]\n        elif len(left_indices) > len(right_indices):\n            left_indices = left_indices[:len(right_indices)]\n        random.shuffle(left_indices)\n        random.shuffle(right_indices)\n        for left_idx, right_idx in zip(left_indices, right_indices):\n            if (right_idx == left_idx) and (left_lang == right_lang):\n                continue\n            if (left_idx, right_idx) in used_pairs:\n                continue\n            used_pairs.add((left_idx, right_idx))\n            used_pairs.add((right_idx, left_idx))\n            text_pairs_and_labels.append(\n                (\n                    texts[left_lang][left_idx][0],\n                    texts[right_lang][right_idx][0],\n                    1\n                )\n            )\n            number_of_samples += 1\n            if number_of_samples >= iterations:\n                break\n        del left_indices, right_indices\n        print('  number of \"1-1\" pairs is {0};'.format(number_of_samples))\n        number_of_samples = 0\n        iterations = (2 * n_samples_for_lang_pair) // 4\n        iterations -= n_samples_for_lang_pair // 4\n        if len(left_negative_indices) > iterations:\n            left_indices = np.random.choice(\n                left_negative_indices,\n                min(iterations * 2, len(left_negative_indices)),\n                p=left_negative_probas, replace=False\n            ).tolist()\n        else:\n            left_indices = left_negative_indices\n        if len(right_negative_indices) > iterations:\n            right_indices = np.random.choice(\n                right_negative_indices,\n                min(iterations * 2, len(right_negative_indices)),\n                p=right_negative_probas, replace=False\n            ).tolist()\n        else:\n            right_indices = right_negative_indices\n        if len(left_indices) < len(right_indices):\n            right_indices = right_indices[:len(left_indices)]\n        elif len(left_indices) > len(right_indices):\n            left_indices = left_indices[:len(right_indices)]\n        random.shuffle(left_indices)\n        random.shuffle(right_indices)\n        for left_idx, right_idx in zip(left_indices, right_indices):\n            if (right_idx == left_idx) and (left_lang == right_lang):\n                continue\n            if (left_idx, right_idx) in used_pairs:\n                continue\n            used_pairs.add((left_idx, right_idx))\n            used_pairs.add((right_idx, left_idx))\n            text_pairs_and_labels.append(\n                (\n                    texts[left_lang][left_idx][0],\n                    texts[right_lang][right_idx][0],\n                    1\n                )\n            )\n            number_of_samples += 1\n            if number_of_samples >= iterations:\n                break\n        del left_indices, right_indices\n        print('  number of \"0-0\" pairs is {0};'.format(number_of_samples))\n        number_of_samples = 0\n        iterations = n_samples_for_lang_pair\n        iterations -= (2 * n_samples_for_lang_pair) // 4\n        if len(left_negative_indices) > iterations:\n            left_indices = np.random.choice(\n                left_negative_indices,\n                min(iterations * 2, len(left_negative_indices)),\n                p=left_negative_probas, replace=False\n            ).tolist()\n        else:\n            left_indices = left_negative_indices\n        if len(right_positive_indices) > iterations:\n            right_indices = np.random.choice(\n                right_positive_indices,\n                min(iterations * 2, len(right_positive_indices)),\n                p=right_positive_probas, replace=False\n            ).tolist()\n        else:\n            right_indices = right_positive_indices\n        if len(left_indices) < len(right_indices):\n            right_indices = right_indices[:len(left_indices)]\n        elif len(left_indices) > len(right_indices):\n            left_indices = left_indices[:len(right_indices)]\n        random.shuffle(left_indices)\n        random.shuffle(right_indices)\n        for left_idx, right_idx in zip(left_indices, right_indices):\n            if (right_idx == left_idx) and (left_lang == right_lang):\n                continue\n            if (left_idx, right_idx) in used_pairs:\n                continue\n            used_pairs.add((left_idx, right_idx))\n            used_pairs.add((right_idx, left_idx))\n            if random.random() >= 0.5:\n                text_pairs_and_labels.append(\n                    (\n                        texts[left_lang][left_idx][0],\n                        texts[right_lang][right_idx][0],\n                        0\n                    )\n                )\n            else:\n                text_pairs_and_labels.append(\n                    (\n                        texts[right_lang][right_idx][0],\n                        texts[left_lang][left_idx][0],\n                        0\n                    )\n                )\n            number_of_samples += 1\n            if number_of_samples >= iterations:\n                break\n        del left_indices, right_indices\n        print('  number of \"0-1\" or \"1-0\" pairs is {0}.'.format(\n            number_of_samples\n        ))\n    random.shuffle(text_pairs_and_labels)\n    n_steps = len(text_pairs_and_labels) // batch_size\n    print('Samples number of the data set is {0}.'.format(\n        len(text_pairs_and_labels)\n    ))\n    print('Samples number per each language pair is {0}.'.format(\n        n_samples_for_lang_pair\n    ))\n    tokens_of_left_texts = tokenize_all(\n        texts=[cur[0] for cur in text_pairs_and_labels],\n        tokenizer=tokenizer, maxlen=maxlen\n    )\n    tokens_of_left_texts = np.array(tokens_of_left_texts, dtype=np.int32)\n    print('')\n    print('3 examples of left texts after tokenization:')\n    for _ in range(3):\n        idx = random.randint(0, len(text_pairs_and_labels) - 1)\n        print('  {0}'.format(text_pairs_and_labels[idx][0]))\n        print('  {0}'.format(tokens_of_left_texts[idx].tolist()))\n        print('')\n    tokens_of_right_texts = tokenize_all(\n        texts=[cur[1] for cur in text_pairs_and_labels],\n        tokenizer=tokenizer, maxlen=maxlen\n    )\n    tokens_of_right_texts = np.array(tokens_of_right_texts, dtype=np.int32)\n    print('3 examples of right texts after tokenization:')\n    for _ in range(3):\n        idx = random.randint(0, len(text_pairs_and_labels) - 1)\n        print('  {0}'.format(text_pairs_and_labels[idx][1]))\n        print('  {0}'.format(tokens_of_right_texts[idx].tolist()))\n        print('')\n    siamese_labels = np.array([cur[2] for cur in text_pairs_and_labels],\n                              dtype=np.int32)\n    print('Number of positive siamese samples is {0} from {1}.'.format(\n        int(sum(siamese_labels)), siamese_labels.shape[0]))\n    err_msg = '{0} != 2'.format(len(tokens_of_left_texts.shape))\n    assert len(tokens_of_left_texts.shape) == 2, err_msg\n    err_msg = '{0} != 1'.format(len(siamese_labels.shape))\n    assert len(siamese_labels.shape) == 1, err_msg\n    err_msg = '{0} != {1}'.format(tokens_of_left_texts.shape, tokens_of_right_texts.shape)\n    assert tokens_of_left_texts.shape == tokens_of_right_texts.shape, err_msg\n    err_msg = '{0} != {1}'.format(tokens_of_left_texts.shape[0], siamese_labels.shape[0])\n    assert tokens_of_left_texts.shape[0] == siamese_labels.shape[0], err_msg\n    if shuffle:\n        err_msg = '{0} is too small number of samples for the data set!'.format(\n            len(text_pairs_and_labels))\n        assert n_steps >= 50, err_msg\n        dataset = tf.data.Dataset.from_tensor_slices(\n            (\n                (\n                    tokens_of_left_texts,\n                    tokens_of_right_texts\n                ),\n                siamese_labels\n            )\n        ).repeat().batch(batch_size)\n    else:\n        dataset = tf.data.Dataset.from_tensor_slices(\n            (\n                (\n                    tokens_of_left_texts,\n                    tokens_of_right_texts\n                ),\n                siamese_labels\n            )\n        ).batch(batch_size)\n    del text_pairs_and_labels\n    return dataset, n_steps","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def build_feature_extractor(...)**\n\nThis function builds a sentence embedder, based on XLM-RoBERTa, as a Keras model."},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_feature_extractor(transformer_name: str, padding: int,\n                            max_len: int) -> tf.keras.Model:\n    xlmroberta_config = XLMRobertaConfig.from_pretrained(transformer_name)\n    max_position_embeddings = xlmroberta_config.max_position_embeddings\n    if max_len > (max_position_embeddings - 2):\n        err_msg = 'max_text_len = {0} is too large! It must be less ' \\\n                  'then {1}.'.format(max_text_len, max_position_embeddings - 1)\n        raise ValueError(err_msg)\n    output_embedding_size = xlmroberta_config.hidden_size\n    word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                     name=\"base_word_ids_FE\")\n    attention_mask = AttentionMaskLayer(\n        pad_token_id=padding, name='base_attention_mask_FE',\n        trainable=False\n    )(word_ids)\n    del xlmroberta_config\n    transformer_layer = TFXLMRobertaModel.from_pretrained(\n        pretrained_model_name_or_path=transformer_name,\n        name='Transformer'\n    )\n    sequence_output = transformer_layer(\n        [word_ids, attention_mask]\n    )[0]\n    output_mask = tf.cast(attention_mask, dtype=tf.bool)\n    pooled_output = tf.keras.layers.GlobalAvgPool1D(\n        name='AvePool_FE'\n    )(sequence_output, mask=output_mask)\n    text_embedding = tf.keras.layers.LayerNormalization(\n        name='Emdedding_FE'\n    )(pooled_output)\n    fe_model = tf.keras.Model(\n        inputs=word_ids,\n        outputs=text_embedding,\n        name='FeatureExtractor'\n    )\n    fe_model.build(input_shape=(None, max_len))\n    return fe_model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def euclidean_distance(vects):\n    x, y = vects\n    sum_square = tf.keras.backend.sum(tf.keras.backend.square(x - y),\n                                      axis=1, keepdims=True)\n    return tf.keras.backend.sqrt(\n        tf.keras.backend.maximum(sum_square, tf.keras.backend.epsilon())\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def eucl_dist_output_shape(shapes):\n    shape1, shape2 = shapes\n    return (shape1[0], 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def build_siamese_nn(...)**\n\nThis function builds a Siamese neural network, which consists of two XLM-RoBERTa sentence embedders with shared weights (the XLM-RoBERTa sentence embedder is created using the *build_feature_extractor* function), a Euclidean distance layer, and a distance-based logistic loss.\n\nWhen I compile my *tf.keras.Model* object, then I set an [Adam](https://arxiv.org/abs/1711.05101) algorithm as optimizer with ."},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_siamese_nn(transformer_name: str, max_len: int, padding: int,\n                     stepsize: int) -> Tuple[tf.keras.Model, tf.keras.Model]:\n    left_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                          name=\"left_word_ids\")\n    right_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype=tf.int32,\n                                           name=\"right_word_ids\")\n    fe_ = build_feature_extractor(\n        transformer_name=transformer_name,\n        padding=padding,\n        max_len=max_len\n    )\n    left_text_embedding = fe_(left_word_ids)\n    right_text_embedding = fe_(right_word_ids)\n    distance_layer = tf.keras.layers.Lambda(\n        function=euclidean_distance,\n        output_shape=eucl_dist_output_shape,\n        name='L2DistLayer'\n    )([left_text_embedding, right_text_embedding])\n    nn = tf.keras.Model(\n        inputs=[left_word_ids, right_word_ids],\n        outputs=distance_layer,\n        name='SiameseXLMR'\n    )\n    lr_schedule = tfa.optimizers.Triangular2CyclicalLearningRate(\n        initial_learning_rate=1e-6,\n        maximal_learning_rate=5e-5,\n        step_size=3 * stepsize\n    )\n    optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule)\n    nn.compile(\n        optimizer=optimizer,\n        loss=DBLLogLoss()\n    )\n    fe_.summary()\n    print('')\n    nn.summary()\n    return nn, fe_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def show_training_process(...)**\n\nThis function shows a training metric curve and a validation one using a Tensorflow log (i.e. the *tf.keras.callbacks.History* object). A kind of metric (loss, accuracy, or any other measure) is specified by the additional parameter *metric_name*."},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_training_process(history: tf.keras.callbacks.History, metric_name: str,\n                          figure_id: int=1):\n    val_metric_name = 'val_' + metric_name\n    err_msg = 'The metric \"{0}\" is not found! Available metrics are: {1}'.format(\n        metric_name, list(history.history.keys()))\n    assert metric_name in history.history, err_msg\n    plt.figure(figure_id, figsize=(5, 5))\n    plt.plot(list(range(len(history.history[metric_name]))),\n             history.history[metric_name], label='Training {0}'.format(metric_name))\n    if val_metric_name in history.history:\n        assert len(history.history[metric_name]) == len(history.history['val_' + metric_name])\n        plt.plot(list(range(len(history.history['val_' + metric_name]))),\n                 history.history['val_' + metric_name], label='Validation {0}'.format(metric_name))\n    plt.xlabel('Epochs')\n    plt.ylabel(metric_name)\n    plt.title('Training process')\n    plt.legend(loc='best')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def train_siamese_nn(...)**\n\nThis function applies a training procedure to a specified Siamese neural network. Two stopping criteria are used at the same time: 1) \"classical\" early stopping criterion; 2) stopping by exceeding of maximal training duration (in seconds). The best weights of the neural network, which correspond to training moment with a minimal value of validation loss, are saved in the special binary file *model_weights_path*."},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_siamese_nn(nn: tf.keras.Model, trainset: tf.data.Dataset, steps_per_trainset: int,\n                     steps_per_epoch: int, validset: tf.data.Dataset, max_duration: int,\n                     model_weights_path: str):\n    assert steps_per_trainset >= steps_per_epoch\n    n_epochs = max(30, int(round(10.0 * steps_per_trainset / float(steps_per_epoch))))\n    print('Maximal duration of the Siamese XLM-R training is {0} '\\\n          'seconds.'.format(max_duration))\n    callbacks = [\n        tf.keras.callbacks.EarlyStopping(patience=9, monitor='val_loss', mode='min',\n                                         restore_best_weights=False, verbose=True),\n        tf.keras.callbacks.ModelCheckpoint(model_weights_path, monitor='val_loss',\n                                           mode='min', save_best_only=True,\n                                           save_weights_only=True, verbose=True),\n        tfa.callbacks.TimeStopping(seconds=max_duration, verbose=True)\n    ]\n    history = nn.fit(\n        trainset,\n        steps_per_epoch=steps_per_epoch,\n        validation_data=validset,\n        epochs=n_epochs,\n        callbacks=callbacks\n    )\n    show_training_process(history, 'loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def calculate_features_of_texts(...)**\n\nThis function calculates sentence embeddings (i.e. fixed-size semantic vectors for sentences) using a trained XLM-RoBERTa-based feature extractor. Input texts with their toxicity labels (or integer identifiers instead of labels) are specified in a format which is like as a result format of the *load_train_set* and the *load_test_set* functions.\n\nThe returned object is a dictionary, where a key is a language and a value is a tuple consists of two NumPy arrays (the first of them is a matrix of sentence vectors, and the second of them is a 1d-array of integer labels or identifiers)."},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_features_of_texts(texts: Dict[str, List[Tuple[str, int]]],\n                                tokenizer: XLMRobertaTokenizer, maxlen: int,\n                                fe: tf.keras.Model, batch_size: int,\n                                max_dataset_size: int = 0) -> \\\n        Tuple[Dict[str, Tuple[np.ndarray, np.ndarray]]]:\n    languages = sorted(list(texts.keys()))\n    datasets_by_languages = dict()\n    if max_dataset_size > 0:\n        max_size_per_lang = max_dataset_size // len(languages)\n        err_msg = '{0} is too small number of dataset samples!'.format(max_dataset_size)\n        assert max_size_per_lang > 0, err_msg\n    else:\n        max_size_per_lang = 0\n    print('Number of languages is {0}.'.format(len(texts)))\n    for cur_lang in languages:\n        print('')\n        print('Language \"{0}\": featurizing is started.'.format(cur_lang))\n        selected_indices = list(range(len(texts[cur_lang])))\n        print('Number of texts is {0}.'.format(len(selected_indices)))\n        if max_size_per_lang > 0:\n            if len(selected_indices) > max_size_per_lang:\n                selected_indices = random.sample(\n                    population=selected_indices,\n                    k=max_size_per_lang\n                )\n        tokens_of_texts = tokenize_all(\n            texts=[texts[cur_lang][idx][0] for idx in selected_indices],\n            tokenizer=tokenizer, maxlen=maxlen\n        )\n        tokens_of_texts = np.array(tokens_of_texts, dtype=np.int32)\n        print('')\n        print('3 examples of texts after tokenization:')\n        for _ in range(3):\n            idx = random.randint(0, len(selected_indices) - 1)\n            print('  {0}'.format(texts[cur_lang][selected_indices[idx]][0]))\n            print('  {0}'.format(tokens_of_texts[idx].tolist()))\n            print('')\n        X = []\n        n_batches = int(np.ceil(len(selected_indices) / float(batch_size)))\n        if n_batches >= 10:\n            n_data_parts = 10\n            data_part_size = int(np.ceil(n_batches / float(n_data_parts)))\n        else:\n            n_data_parts = 0\n        data_part_counter = 0\n        for batch_idx in range(n_batches):\n            batch_start = batch_idx * batch_size\n            batch_end = min(len(selected_indices), batch_start + batch_size)\n            res = fe.predict_on_batch(tokens_of_texts[batch_start:batch_end])\n            if not isinstance(res, np.ndarray):\n                res = res.numpy()\n            X.append(res)\n            del res\n            if n_data_parts > 0:\n                if (batch_idx + 1) % data_part_size == 0:\n                    data_part_counter += 1\n                    print('  {0}% of texts are featured.'.format(data_part_counter * 10))\n        if (n_data_parts > 0) and (data_part_counter < n_data_parts):\n            print('  100% of texts are featured.')\n        X = np.vstack(X)\n        y = np.array([texts[cur_lang][idx][1] for idx in selected_indices], dtype=np.int32)\n        datasets_by_languages[cur_lang] = (X, y)\n        del X, y, selected_indices\n        print('Language \"{0}\": featurizing is finished.'.format(cur_lang))\n    return datasets_by_languages","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def generate_featured_data(...)**\n\nThis function prepares all generated feature vectors for training and submission in a form that is more comfortable for future experiments with various classifiers in the new feature space. The result of the function is a 3-element tuple:\n\n1. training data as two NumPy arrays: matrix of feature vectors for the training of a final classifier and vector of corresponded toxicity labels;\n2. splitting of training data by languages as a dictionary of train/test indices by language name (so, one cross-validation fold corresponds to one language);\n3. data for submitting as two NumPy arrays: matrix of feature vectors as inputs for the final classifier and vector of corresponded submission samples identifiers."},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_featured_data(\n    features_by_lang: Dict[str, Tuple[np.ndarray, np.ndarray]],\n    features_for_submission: Union[Dict[str, Tuple[np.ndarray, np.ndarray]], None] = None,\n) -> Tuple[Tuple[np.ndarray, np.ndarray], Dict[str, Tuple[np.ndarray, np.ndarray]], \\\n           Union[Tuple[np.ndarray, np.ndarray]], None]:\n    X_embedded = []\n    y_embedded = []\n    split_by_languages = dict()\n    start_pos = 0\n    for cur_lang in features_by_lang:\n        X_embedded.append(features_by_lang[cur_lang][0])\n        y_embedded.append(features_by_lang[cur_lang][1])\n        split_by_languages[cur_lang] = (\n            set(),\n            set(range(start_pos, start_pos + features_by_lang[cur_lang][1].shape[0]))\n        )\n        start_pos = start_pos + features_by_lang[cur_lang][1].shape[0]\n    featured_data_for_training = (\n        np.vstack(X_embedded),\n        np.concatenate(y_embedded)\n    )\n    del X_embedded, y_embedded\n    err_msg = '{0} != {1}'.format(featured_data_for_training[0].shape[0],\n                                  featured_data_for_training[1].shape[0])\n    assert featured_data_for_training[0].shape[0] == featured_data_for_training[1].shape[0], err_msg\n    for cur_lang in features_by_lang:\n        indices_for_testing = split_by_languages[cur_lang][1]\n        indices_for_training = set(range(featured_data_for_training[0].shape[0]))\n        indices_for_training -= indices_for_testing\n        split_by_languages[cur_lang] = (\n            np.array(sorted(list(indices_for_training)), dtype=np.int32),\n            np.array(sorted(list(indices_for_testing)), dtype=np.int32)\n        )\n        del indices_for_training, indices_for_testing\n    all_languages = sorted(list(split_by_languages.keys()))\n    prev_lang = all_languages[0]\n    assert len(set(split_by_languages[prev_lang][1].tolist()) & \\\n               set(split_by_languages[prev_lang][0].tolist())) == 0\n    for cur_lang in all_languages[1:]:\n        assert len(set(split_by_languages[cur_lang][1].tolist()) & \\\n                   set(split_by_languages[cur_lang][0].tolist())) == 0\n        assert len(set(split_by_languages[cur_lang][1].tolist()) & \\\n                   set(split_by_languages[prev_lang][1].tolist())) == 0\n        prev_lang = cur_lang\n    if features_for_submission is None:\n        return featured_data_for_training, split_by_languages\n    featured_inputs_for_submission = []\n    identifies_for_submission = []\n    for cur_lang in features_for_submission:\n        featured_inputs_for_submission.append(features_for_submission[cur_lang][0])\n        identifies_for_submission.append(features_for_submission[cur_lang][1])\n    featured_data_for_submission = (\n        np.vstack(featured_inputs_for_submission),\n        np.concatenate(identifies_for_submission)\n    )\n    del featured_inputs_for_submission, identifies_for_submission\n    n_samples_for_submission = featured_data_for_submission[0].shape[0]\n    n_IDs_for_submission = featured_data_for_submission[1].shape[0]\n    err_msg = '{0} != {1}'.format(n_samples_for_submission, n_IDs_for_submission)\n    assert n_samples_for_submission == n_IDs_for_submission, err_msg\n    return (featured_data_for_training, split_by_languages, featured_data_for_submission)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**def calculate_projections(...)**\n\nThis function is needed to calculate T-SNE projections of labeled data and their visualization on the 2d space."},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_projections(labeled_data: Tuple[np.ndarray, np.ndarray],\n                          additional_title: str):\n    X_prj = labeled_data[0]\n    y_prj = labeled_data[1]\n    assert len(X_prj.shape) == 2\n    assert len(y_prj.shape) == 1\n    n_samples = X_prj.shape[0]\n    err_msg = '{0} != {1}'.format(n_samples, y_prj.shape[0])\n    assert n_samples == y_prj.shape[0], err_msg\n    if n_samples > 3000:\n        test_size = 1500.0 / float(n_samples)\n        _, X_prj, _, y_prj = train_test_split(X_prj, y_prj, test_size=test_size,\n                                              random_state=42, stratify=y_prj)\n    X_prj = TSNE(n_components=2, n_jobs=-1).fit_transform(X_prj)\n    plt.figure(figsize=(10, 10))\n    indices_of_negative_classes = list(filter(\n        lambda sample_idx: y_prj[sample_idx] == 0,\n        range(y_prj.shape[0])\n    ))\n    xy = X_prj[indices_of_negative_classes]\n    plt.plot(xy[:, 0], xy[:, 1], 'o', color='g', markersize=4,\n             label='Normal texts')\n    indices_of_positive_classes = list(filter(\n        lambda sample_idx: y_prj[sample_idx] > 0,\n        range(y_prj.shape[0])\n    ))\n    xy = X_prj[indices_of_positive_classes]\n    plt.plot(xy[:, 0], xy[:, 1], 'o', color='r', markersize=6,\n             label='Toxic texts')\n    if len(additional_title) > 0:\n        if additional_title[0].isalnum():\n            plt.title('Toxic and normal texts {0}'.format(additional_title))\n        else:\n            plt.title('Toxic and normal texts{0}'.format(additional_title))\n    else:\n        plt.title('Toxic and normal texts')\n    plt.legend(loc='best')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Declaration of all functions is finished, and now I start to write the main code"},{"metadata":{},"cell_type":"markdown","source":"I fix start time moment of experiments"},{"metadata":{"trusted":true},"cell_type":"code","source":"experiment_start_time = time.time()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I detect a hardware for my experiments (GPU or TPU) and create a corresponded [distribution strategy](https://www.tensorflow.org/guide/distributed_training) as a special Tensorflow object."},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n    physical_devices = tf.config.list_physical_devices('GPU')\n    for device_idx in range(strategy.num_replicas_in_sync):\n        tf.config.experimental.set_memory_growth(physical_devices[device_idx], True)\nmodel_name = 'jplu/tf-xlm-roberta-base'\nmax_seq_len = 128\nbatch_size_for_siamese = 32 * strategy.num_replicas_in_sync\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\nprint('Model name: {0}'.format(model_name))\nprint('Maximal length of sequence is {0}'.format(max_seq_len))\nprint('Batch size for the Siamese XLM-RoBERTa is {0}'.format(\n    batch_size_for_siamese))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I initialize a seed for all pseudo-random generators. This thing is very important for an experiment reproducibility!"},{"metadata":{"trusted":true},"cell_type":"code","source":"random.seed(42)\nnp.random.seed(42)\ntf.random.set_seed(42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I set all file paths to the input data and to all generated results, i.e. trained Siamese XLM-R and featured data for competition, calculated using this model."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_dir = '/kaggle/input/jigsaw-multilingual-toxic-comment-classification'\ntmp_roberta_name = '/kaggle/working/siamese_xlmr.h5'\nfeature_extractor_name = '/kaggle/working/xlmr_fe.h5'\ntmp_features_name = '/kaggle/working/features_by_siamese_xlmr.pkl'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I download meta-information about selected XLM-R from the Hugginface Transformers, and I prepare the configuration and tokenizer accordingly to this meta-information."},{"metadata":{"trusted":true},"cell_type":"code","source":"xlmroberta_tokenizer = AutoTokenizer.from_pretrained(model_name)\nxlmroberta_config = XLMRobertaConfig.from_pretrained(model_name)\nprint(xlmroberta_config)\nprint('xlmroberta_tokenizer.pad_token_id',\n      xlmroberta_tokenizer.pad_token_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I detect a sentence embedding size from the XLM-R configuration, and I check a maximal length of token sequence accordingly to this configuration."},{"metadata":{"trusted":true},"cell_type":"code","source":"sentence_embedding_size = xlmroberta_config.hidden_size\nprint('Sentence embedding size is {0}'.format(sentence_embedding_size))\nassert max_seq_len <= xlmroberta_config.max_position_embeddings","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I load data for training. These data must contain labeled English texts."},{"metadata":{"trusted":true},"cell_type":"code","source":"corpus_for_training = load_train_set(\n    os.path.join(dataset_dir, \"jigsaw-unintended-bias-train.csv\"),\n    text_field=\"comment_text\", lang_field=\"lang\",\n    sentiment_fields=[\"toxic\", \"severe_toxicity\", \"obscene\", \"identity_attack\",\n                      \"insult\", \"threat\"]\n)\nassert 'en' in corpus_for_training","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I load multilingual data for training in addition to the abovementioned English data. These data must represent texts in three languages (non-English only)"},{"metadata":{"trusted":true},"cell_type":"code","source":"multilingual_corpus = load_train_set(\n    os.path.join(dataset_dir, \"validation.csv\"),\n    text_field=\"comment_text\", lang_field=\"lang\", sentiment_fields=[\"toxic\", ]\n)\nassert 'en' not in multilingual_corpus\nmax_size = 0\nprint('Multilingual data:')\nfor language in sorted(list(multilingual_corpus.keys())):\n    print('  {0}\\t\\t{1} samples'.format(language, len(multilingual_corpus[language])))\n    assert set(map(lambda cur: cur[1], multilingual_corpus[language])) == {0, 1}\n    if len(multilingual_corpus[language]) > max_size:\n        max_size = len(multilingual_corpus[language])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I split the text corpus in English into three parts. The first part will rich the multilingual corpus for training (as the fourth language). The second part will be used as a data source for a Siamese network validation to implement early stopping. Finally, the third part, which is the largest, will be used as a data source for the training set of a Siamese network."},{"metadata":{"trusted":true},"cell_type":"code","source":"err_msg = 'Size of English corpus = {0} is too small!'.format(\n    len(corpus_for_training['en'])\n)\nassert len(corpus_for_training['en']) >= (max_size * 10), err_msg\nrandom.shuffle(corpus_for_training['en'])\nmultilingual_corpus['en'] = corpus_for_training['en'][:max_size]\nn_validation = int(round(0.1 * (len(corpus_for_training['en']) - max_size)))\ncorpus_for_validation = {'en': corpus_for_training['en'][max_size:(max_size + n_validation)]}\ncorpus_for_training = {'en': corpus_for_training['en'][(n_validation + max_size):]}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I load multilingual data for final toxicity classification and submitting to the competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"texts_for_submission = load_test_set(\n    os.path.join(dataset_dir, \"test.csv\"),\n    text_field=\"content\", lang_field=\"lang\", id_field=\"id\"\n)\nprint('Data for submission:')\nfor language in sorted(list(texts_for_submission.keys())):\n    print('  {0}\\t\\t{1} samples'.format(language, len(texts_for_submission[language])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I prepare a dataset for the Siamese XLM-R training (the number of labeled pairs in this dataset is equal to 500000)."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_for_training, n_batches_per_data = build_siamese_dataset(\n    texts=corpus_for_training, dataset_size=500000,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    batch_size=batch_size_for_siamese, shuffle=True\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I prepare a dataset for the Siamese XLM-R validation during training (the number of labeled pairs in this dataset is equal to 1000)."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_for_validation, n_batches_per_epoch = build_siamese_dataset(\n    texts=corpus_for_validation, dataset_size=1000,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    batch_size=batch_size_for_siamese, shuffle=False\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I specify optimal number of steps (mini-batches) per single training epoch as value which is less than full number of mini-batches in the training data, because waiting for full training set processing per epoch is not rational."},{"metadata":{"trusted":true},"cell_type":"code","source":"steps_per_single_epoch = min(20 * n_batches_per_epoch, n_batches_per_data)\nprint('Number of steps (mini-batches) per single epoch is {0}.'.format(\n    steps_per_single_epoch\n))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete all data which becomes unnecessary and call the garbage collector."},{"metadata":{"trusted":true},"cell_type":"code","source":"del corpus_for_training, corpus_for_validation\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I fix the duration of all preparing procedures, implemented in the abovementioned code."},{"metadata":{"trusted":true},"cell_type":"code","source":"preparing_end_time = time.time()\npreparing_duration = int(round(preparing_end_time - experiment_start_time))\nprint(\"Duration of data loading and preparing to the Siamese NN training is \"\n      \"{0} seconds.\".format(preparing_duration))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I build the Siamese XLM-R."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    siamese_network, feature_extractor = build_siamese_nn(\n        transformer_name=model_name,\n        max_len=max_seq_len,\n        padding=xlmroberta_tokenizer.pad_token_id,\n        stepsize=steps_per_single_epoch\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I calculate the featured training set for the final classifier using an untrained XLM-R-based feature extractor. In reality, these data need not for any classifier training, but they need for their 2d projections representation."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_for_training_ = calculate_features_of_texts(\n    texts=multilingual_corpus,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    fe=feature_extractor,\n    batch_size=batch_size_for_siamese\n)\nassert len(dataset_for_training_) == 4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I simplify the abovementioned data by means of their transforming from a dictionary by languages to a \"normal\" NumPy arrays pair (*X* and *y*)."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_for_cls_training, _ = generate_featured_data(dataset_for_training_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete all data which becomes unnecessary and call the garbage collector."},{"metadata":{"trusted":true},"cell_type":"code","source":"del dataset_for_training_\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I show projections of data, which are featured by an untrained XLM-R."},{"metadata":{"trusted":true},"cell_type":"code","source":"calculate_projections(data_for_cls_training, 'before XLM-R training')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete all data which becomes unnecessary and call the garbage collector."},{"metadata":{"trusted":true},"cell_type":"code","source":"del data_for_cls_training\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I fix the duration of all procedures for data projecting."},{"metadata":{"trusted":true},"cell_type":"code","source":"projecting_duration = int(round(time.time() - preparing_end_time))\nprint(\"Duration of data projection is {0} seconds.\".format(projecting_duration))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lastly, I train my Siamese XLM-R."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_siamese_nn(nn=siamese_network, trainset=dataset_for_training,\n                 steps_per_trainset=n_batches_per_data,\n                 steps_per_epoch=steps_per_single_epoch,\n                 validset=dataset_for_validation,\n                 max_duration=int(round(\n                     3600 * 2.5 - preparing_duration - 3.0 * projecting_duration\n                 )),\n                 model_weights_path=tmp_roberta_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete all data which becomes unnecessary and call the garbage collector."},{"metadata":{"trusted":true},"cell_type":"code","source":"del dataset_for_training\ndel dataset_for_validation\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I load the best weights of XLM-R, found during the training process, from the HDF5 binary data file."},{"metadata":{"trusted":true},"cell_type":"code","source":"siamese_network.load_weights(tmp_roberta_name)\nos.remove(tmp_roberta_name)\nfeature_extractor.save_weights(feature_extractor_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete the Siamese XLM-R (but not the XLM-R-based feature extractor!), and after that, I call the garbage collector."},{"metadata":{"trusted":true},"cell_type":"code","source":"del siamese_network\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I calculate the featured training set for the final classifier using a trained XLM-R-based feature extractor."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_for_training_ = calculate_features_of_texts(\n    texts=multilingual_corpus,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    fe=feature_extractor,\n    batch_size=batch_size_for_siamese\n)\nassert len(dataset_for_training_) == 4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also, I calculate the featured data set, which will be used to submit predictions to the competition using some final classifier, trained on the abovementioned featured training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_for_submission_ = calculate_features_of_texts(\n    texts=texts_for_submission,\n    tokenizer=xlmroberta_tokenizer, maxlen=max_seq_len,\n    fe=feature_extractor,\n    batch_size=batch_size_for_siamese\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete the XLM-R-based feature extractor and corresponded tokenizer. They are no longer needed."},{"metadata":{"trusted":true},"cell_type":"code","source":"del feature_extractor, xlmroberta_tokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I simplify the abovementioned data for the final classifier training and submitting using this classifier. Simplification is realized by means of these data transforming from a dictionary by languages to a \"normal\" NumPy arrays pairs (X and y). Also, in this way, I get a CV splitting of training data."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_for_cls_training, cv_splitting, submission_data_for_cls = generate_featured_data(\n    dataset_for_training_,\n    dataset_for_submission_\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I delete all data which becomes unnecessary and call the garbage collector."},{"metadata":{"trusted":true},"cell_type":"code","source":"del dataset_for_training_, dataset_for_submission_\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I show projections of data, which are featured by a trained XLM-R."},{"metadata":{"trusted":true},"cell_type":"code","source":"calculate_projections(data_for_cls_training, 'after training of XLM-R as a Siamese network')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"At last, I save all featured data in the special binary file. After that, I can re-use these data in any experiment with the final classifier. The Siamese XLM-R is no longer needed because it had done its work!"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(tmp_features_name, 'wb') as fp:\n    pickle.dump(\n        obj=(data_for_cls_training, cv_splitting, submission_data_for_cls),\n        file=fp,\n        protocol=pickle.HIGHEST_PROTOCOL\n    )","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}