{"cells":[{"metadata":{},"cell_type":"markdown","source":"## About this notebook\n\n*[Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)* is the 3rd annual competition organized by the Jigsaw team. It follows *[Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)*, the original 2018 competition, and *[Jigsaw Unintended Bias in Toxicity Classification](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification)*, which required the competitors to consider biased ML predictions in their new models. This year, the goal is to use english only training data to run toxicity predictions on many different languages, which can be done using multilingual models, and speed up using TPUs.\n\nMany awesome notebooks has already been made so far. Many of them used really cool technologies like [Pytorch XLA](https://www.kaggle.com/theoviel/bert-pytorch-huggingface-starter). This notebook instead aims at constructing a **fast, concise, reusable, and beginner-friendly model scaffold**. It will focus on the following points:\n* **Using Tensorflow and Keras**: Tensorflow is a powerful framework, and Keras makes the training process extremely easy to understand. This is especially good for beginners to learn how to use TPUs, and for experts to focus on the modelling aspect.\n* **Using Huggingface's `transformers` library**: [This library](https://huggingface.co/transformers/) is extremely popular, so using this let you easily integrate the end result into your ML pipelines, and can be easily reused for your other projects.\n* **Native TPU usage**: The TPU usage is abstracted using the native `strategy` that was created using Tensorflow's `tf.distribute.experimental.TPUStrategy`. This avoids getting too much into the lower-level aspect of TPU management.\n* **Use a subset of the data**: Instead of using the entire dataset, we will only use the 2018 subset of the data available, which makes this much faster, all while achieving a respectable accuracy."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport warnings\n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\nfrom kaggle_datasets import KaggleDatasets\nimport transformers\nimport traitlets\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom tqdm.notebook import tqdm\nfrom tokenizers import BertWordPieceTokenizer\nfrom sklearn.metrics import roc_auc_score\n\nwarnings.simplefilter(\"ignore\")\n\nnp.random.seed(100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=256, maxlen=512):\n    tokenizer.enable_truncation(max_length=maxlen)\n    tokenizer.enable_padding(max_length=maxlen)\n    all_ids = []\n    \n    for i in tqdm(range(0, len(texts), chunk_size)):\n        text_chunk = texts[i:i+chunk_size].tolist()\n        encs = tokenizer.encode_batch(text_chunk)\n        all_ids.extend([enc.ids for enc in encs])\n    \n    return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(transformer, loss='binary_crossentropy', max_len=512):\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    emb = transformer(input_word_ids)[0]\n    _avg = tf.keras.layers.GlobalAveragePooling1D()(emb)\n    _max = tf.keras.layers.GlobalMaxPooling1D()(emb)\n    x = tf.keras.layers.Concatenate()([_avg, _max])\n    x = tf.keras.layers.Dropout(0.15)(x)\n    out = Dense(1, activation='sigmoid')(x)\n    \n    model = Model(inputs=input_word_ids, outputs=out)\n    model.compile(Adam(lr=3e-5), loss=loss, metrics=[tf.keras.metrics.AUC()])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU Configs"},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# Create strategy from tpu\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(tpu)\ntf.tpu.experimental.initialize_tpu_system(tpu)\nstrategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n# Data access\n#GCS_DS_PATH = KaggleDatasets().get_gcs_path('kaggle/input/') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create fast tokenizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = transformers.BertTokenizer.from_pretrained('bert-base-multilingual-uncased')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load text data into memory"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lang_densety = test.groupby('lang').count()['id']/ len(test)\ntarget_densety = valid.groupby('toxic').count()['id']/ len(valid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lang_densety","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_densety","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"N_SAMPLES = 500000\n\ntrain_dfs = []\nfor lang in ['es', 'it', 'pt', 'tr', 'ru', 'fr']:\n    _df = pd.read_csv(f\"/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-{lang}-cleaned.csv\")\n    \n    _df0 = _df.loc[_df.toxic == 0, :]\n    _df1 = _df.loc[_df.toxic == 1, :]\n    \n    n_samples = int(N_SAMPLES * lang_densety[lang])\n    \n    n_samples_0 = int(n_samples * target_densety[0])\n    n_samples_1 = int(n_samples * target_densety[1])\n    \n    _df0 = _df0.sample(n_samples_0)\n    _df1 = _df1.sample(n_samples_1)\n    \n    train_dfs.append(pd.concat([_df0, _df1], ignore_index=True).sample(n_samples_0 + n_samples_1))\n    \ntrain = pd.concat(train_dfs, ignore_index=True)\ntrain = train.sample(min(N_SAMPLES, len(train)))\ntrain.toxic = train.toxic.round().astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('toxic').count()['id']/ len(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class TextTransformation:\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        raise NotImplementedError('Abstarct')\n        \nclass LowerCaseTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        return text.lower(), lang\n    \nclass PunctuationTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for p in '?!.,\"#$%\\'()*+-/:;<=>@[\\\\]^_`{|}~' + '“”’' +\"/-'\" + \"&\" + \"¡¿\":\n            if '’' in text:\n                text = text.replace('’', f' \\' ')\n                \n            if '’' in text:\n                text = text.replace('’', f' \\' ')\n              \n            if '—' in text:\n                text = text.replace('—', f' - ')\n                \n            if '–' in text:\n                text = text.replace('–', f' - ')   \n              \n            if '“' in text:\n                text = text.replace('“', f' \" ')   \n                \n            if '«' in text:\n                text = text.replace('«', f' \" ')   \n                \n            if '»' in text:\n                text = text.replace('»', f' \" ')   \n            \n            if '”' in text:\n                text = text.replace('”', f' \" ') \n                \n            if '`' in text:\n                text = text.replace('`', f' \\' ')              \n\n            text = text.replace(p, f' {p} ')\n                \n        return text.strip(), lang\n    \nclass NumericTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in range(10):\n            text = text.replace(str(i), f' {str(i)} ')\n        return text, lang\n    \nclass ETransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['\\u00E8', '\\u00E9', '\\u00EA', '\\u00EB', '\\u0450', '\\u0451']:\n            text = text.replace(i, 'e')\n        return text, lang\n    \nclass ATransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['à', 'á', '\\u00E2', '\\u00E3', '\\u00E4', '\\u00E5']:\n            text = text.replace(i, 'a')\n        return text, lang\n    \nclass OTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['ó', 'ò', 'ö','õ', 'ô']:\n            text = text.replace(i, 'o')\n        return text, lang\n    \n\nclass CTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['ç']:\n            text = text.replace(i, 'c')\n        return text, lang\n    \nclass ITransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['í', 'ı', 'ì']:\n            text = text.replace(i, 'i')\n        return text, lang\n    \nclass STransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['ş']:\n            text = text.replace(i, 's')\n        return text, lang\n    \n    \nclass NTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['ñ', 'n']:\n            text = text.replace(i, 'n')\n        return text, lang\n    \n    \nclass UTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['ù', 'ü', 'û', 'ú']:\n            text = text.replace(i, 'u')\n        return text, lang\n    \nclass GTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['ğ']:\n            text = text.replace(i, 'g')\n        return text, lang\n\nclass RTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in ['r']:\n            text = text.replace(i, 'r')\n        return text, lang\n    \nclass WikiTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        text = text.replace('wikiproject', ' wiki project ')\n        for i in [' vikipedi ', ' wiki ', ' википедии ', \" вики \", ' википедия ', ' viki ', ' wikipedien ', ' википедию ']:\n            text = text.replace(i, ' wikipedia ')\n        return text, lang\n    \nclass PixelTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for i in [' px ']:\n            text = text.replace(i, ' pixel ')\n        return text, lang\n    \n    \nclass RuTransformation(TextTransformation):\n    def __call__(self, text: str, lang: str = None) -> tuple:\n        if lang is not None and lang == 'ru' and 'http' not in text and 'jpg' not in text and 'wikipedia' not in text:\n            text = text.replace('t', 'т')\n            text = text.replace('h', 'н')\n            text = text.replace('b', 'в')\n            text = text.replace('c', 'c')\n            text = text.replace('k', 'к')\n            text = text.replace('e', 'е')\n            text = text.replace('a', 'а')\n        return text, lang\n    \n    \nclass CombineTransformation(TextTransformation):\n    def __init__(self, transformations: list):\n        self._transformations = transformations\n        \n    def __call__(self, text: str, lang: str = None) -> tuple:\n        for transformation in self._transformations:\n            text, lang = transformation(text, lang)\n        return text, lang\n    \n    def append(self, transformation: TextTransformation):\n        self._transformations.append(transformation)\n        \n        \ntransformer = CombineTransformation(\n    [\n        LowerCaseTransformation(),\n        PunctuationTransformation(),\n        NumericTransformation(),\n        ETransformation(),\n        ATransformation(),\n        OTransformation(),\n        CTransformation(),\n        ITransformation(),\n        STransformation(),\n        UTransformation(),\n        GTransformation(),\n        NTransformation(),\n        WikiTransformation(),\n        PixelTransformation(),\n        RuTransformation()\n    ]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['comment_text'] = [v[0] for v in train.apply(lambda x: transformer(x.comment_text), axis=1).values]\nvalid['comment_text'] = [v[0] for v in valid.apply(lambda x: transformer(x.comment_text, x.lang), axis=1).values]\ntest['content'] = [v[0] for v in test.apply(lambda x: transformer(x.content, x.lang), axis=1).values]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sentences = train[\"comment_text\"].apply(lambda x: x.split()).values.tolist() + valid[\"comment_text\"].apply(lambda x: x.split()).values.tolist() + test['content'].apply(lambda x: x.split()).values.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import operator \nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ndef unknown_plot(data):\n    fig, axes = plt.subplots(ncols=1, figsize=(10, 20))\n    plt.tight_layout()\n    \n    sns.barplot(y=list(data.keys()), x=list(data.values()), ax=axes, color='green')\n\n    axes.spines['right'].set_visible(False)\n    axes.set_xlabel('')\n    axes.set_ylabel('')\n    axes.tick_params(axis='x', labelsize=13)\n    axes.tick_params(axis='y', labelsize=13)\n\n    axes.set_title(f'Most unknown tokens', fontsize=15)\n\n    plt.show()\n    \ndef check_coverage(vocab,embeddings_index):\n    a = {}\n    oov = {}\n    k = 0\n    i = 0\n    for word in tqdm(vocab):\n        try:\n            a[word] = embeddings_index[word]\n            k += vocab[word]\n        except:\n\n            oov[word] = vocab[word]\n            i += vocab[word]\n            pass\n\n    print('Found embeddings for {:.2%} of vocab'.format(len(a) / len(vocab)))\n    print('Found embeddings for  {:.2%} of all text'.format(k / (k + i)))\n    sorted_x = sorted(oov.items(), key=operator.itemgetter(1))[::-1]\n\n    return sorted_x\n\n\ndef build_vocab(sentences, verbose =  True):\n    \"\"\"\n    :param sentences: list of list of words\n    :return: dictionary of words and their count\n    \"\"\"\n    vocab = {}\n    for sentence in tqdm(sentences, disable = (not verbose)):\n        for word in sentence:\n            try:\n                vocab[word] += 1\n            except KeyError:\n                vocab[word] = 1\n    return vocab\n\n\nvocab = build_vocab(sentences)\nunknown_vocab = check_coverage(vocab, tokenizer.get_vocab())\nlen(unknown_vocab)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unknown_plot({k:v for i, (k,v) in enumerate(unknown_vocab) if i < 100})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del sentences","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save the loaded tokenizer locally\nsave_path = '/kaggle/working/distilbert_base_uncased/'\nif not os.path.exists(save_path):\n    os.makedirs(save_path)\ntokenizer.save_pretrained(save_path)\n\n# Reload it with the huggingface tokenizers library\nfast_tokenizer = BertWordPieceTokenizer('distilbert_base_uncased/vocab.txt', lowercase=True)\nfast_tokenizer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fast encode"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = fast_encode(train.comment_text.astype(str), fast_tokenizer, maxlen=512)\nx_valid = fast_encode(valid.comment_text.astype(str), fast_tokenizer, maxlen=512)\nx_test = fast_encode(test.content.astype(str), fast_tokenizer, maxlen=512)\ny_train = train.toxic.values\ny_valid = valid.toxic.values\n\ndel train_dfs, train, valid, test\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Build datasets objects"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .repeat()\n    .shuffle(2048)\n    .batch(64)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(64)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(64)\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Focal Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras import backend as K\n\ndef focal_loss(gamma=2., alpha=.15):\n    def focal_loss_fixed(y_true, y_pred):\n        pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n        pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n        return -K.mean(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) - K.mean((1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n    return focal_loss_fixed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load model into the TPU"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nwith strategy.scope():\n    transformer_layer = transformers.TFBertModel.from_pretrained('bert-base-multilingual-uncased')\n    model = build_model(transformer_layer, loss=focal_loss(gamma=1.5), max_len=512)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## RocAuc Callback"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import Callback \n\nclass RocAucCallback(Callback):\n    def __init__(self, test_data, score_thr):\n        self.test_data = test_data\n        self.score_thr = score_thr\n        self.test_pred = []\n        \n    def on_epoch_end(self, epoch, logs=None):\n        if logs['val_auc'] > self.score_thr:\n            print('\\nRun TTA...')\n            self.test_pred.append(self.model.predict(self.test_data))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LrScheduler"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_lrfn(lr_start=0.000001, lr_max=0.000004, \n               lr_min=0.0000001, lr_rampup_epochs=5, \n               lr_sustain_epochs=3, lr_exp_decay=.87):\n    lr_max = lr_max * strategy.num_replicas_in_sync\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) * lr_exp_decay**(epoch - lr_rampup_epochs - lr_sustain_epochs) + lr_min\n        return lr\n    \n    return lrfn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(10, 7))\n\n_lrfn = build_lrfn()\nplt.plot([i for i in range(35)], [_lrfn(i) for i in range(35)]);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train Model"},{"metadata":{},"cell_type":"markdown","source":"### First Stage"},{"metadata":{"trusted":true},"cell_type":"code","source":"lrfn = build_lrfn()\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\ner = tf.keras.callbacks.EarlyStopping(monitor='val_auc', patience=20, restore_best_weights=True, mode='max')\n\ntrain_history = model.fit(\n    train_dataset,\n    steps_per_epoch=200,\n    validation_data=valid_dataset,\n    callbacks=[lr_schedule, er],\n    epochs=35\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bert_weights = model.get_weights()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Second Stage"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(10, 7))\n\nlrfn = build_lrfn(lr_start=0.000001, lr_max=0.000001, \n               lr_min=0.0000001, lr_rampup_epochs=2, \n               lr_sustain_epochs=1, lr_exp_decay=.65)\nplt.plot([i for i in range(15)], [lrfn(i) for i in range(15)]);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split, StratifiedKFold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5)\n\ntest_preds = np.zeros((len(x_test),))\n\nfor tr_idx, vl_idx in skf.split(x_valid, y_valid):\n    lr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=1)\n    er = tf.keras.callbacks.EarlyStopping(monitor='val_auc', patience=20, restore_best_weights=True, mode='max')\n    x_tr, x_val, y_tr, y_val = x_valid[tr_idx], x_valid[vl_idx], y_valid[tr_idx], y_valid[vl_idx]\n\n    train_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices((x_tr, y_tr))\n        .repeat()\n        .shuffle(2048)\n        .batch(32)\n        .prefetch(AUTO)\n    )\n\n    valid_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices((x_val, y_val))\n        .batch(32)\n        .cache()\n        .prefetch(AUTO)\n    )\n    \n    model.set_weights(bert_weights)\n    train_history = model.fit(\n        train_dataset,\n        steps_per_epoch=40,\n        validation_data=valid_dataset,\n        callbacks=[lr_schedule, er],\n        epochs=15)\n    test_preds += model.predict(test_dataset)[:, 0] / 5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['toxic'] = test_preds\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reference\n* [Jigsaw TPU: DistilBERT with Huggingface and Keras](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras)\n* [inference of bert tpu model ml w/ validation](https://www.kaggle.com/abhishek/inference-of-bert-tpu-model-ml-w-validation)\n* [Overview of Text Similarity Metrics in Python](https://towardsdatascience.com/overview-of-text-similarity-metrics-3397c4601f50)\n* [test-en-df](https://www.kaggle.com/bamps53/test-en-df)\n* [val_en_df](https://www.kaggle.com/bamps53/val-en-df)\n* [Jigsaw multilingual toxic - test translated](https://www.kaggle.com/kashnitsky/jigsaw-multilingual-toxic-test-translated)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"0265fa838c634a849c25f9b1762500dc":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"06dba3f6f8904669bff76540450741c1":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"125be473610c43c08dde4e8139e504a4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_3d9b87e0af00442c91c17c573598ee72","placeholder":"​","style":"IPY_MODEL_a6e2ac8a7ebe4ac6ae1f3bc5a90dd6aa","value":" 250/250 [00:19&lt;00:00, 12.88it/s]"}},"17ffee35f43b4c86be709e5512b1637d":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"1a8b9ac751aa4ed7ac3738dc8ef192b1":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"23cf72f5be9548b4a4179286dfbd8a8a":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"249b02767e1d4fc982eb8b009170ea2d":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"Downloading: 100%","description_tooltip":null,"layout":"IPY_MODEL_e881aaefb31843d0857eb62a7066c9d2","max":569,"min":0,"orientation":"horizontal","style":"IPY_MODEL_e32dbfb98ad946c3aa20049cc4aaaf38","value":569}},"2f89af4eb67540e889c1f0ad0992eaed":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"3d9b87e0af00442c91c17c573598ee72":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"3feb1011761142f39f245d4e5e5b367d":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"4416a6602ef743f3a6be716663f30b15":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_7558649d528343aa8add50f9df7936b7","IPY_MODEL_493298ff36fb4121936c96a1545f6a0a"],"layout":"IPY_MODEL_e10014a15736446cbb489b47d0712f7b"}},"493298ff36fb4121936c96a1545f6a0a":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_76083bf6a73b453fa990e4afef82a7d0","placeholder":"​","style":"IPY_MODEL_d800cd3318b64aae87c99b3e3fe26fa3","value":" 32/32 [00:05&lt;00:00,  5.84it/s]"}},"4d0025e1d8b44b7d92741efbc82b998f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"503897d0e3c54e8a90a0c2c2b013983a":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"575f00f44a2b41949bcad32a90946707":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"100%","description_tooltip":null,"layout":"IPY_MODEL_7d8596c77dc84ee18860aad7f615735e","max":1954,"min":0,"orientation":"horizontal","style":"IPY_MODEL_23cf72f5be9548b4a4179286dfbd8a8a","value":1954}},"5ac8b2edfc19497e8399eaee41c3d674":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"689c7ac5dbb244f59366457954e26a4a":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_b595f2bf32974bfda9164f3e59359b5a","IPY_MODEL_e174a5e3402744dbbd4385d9ceccc4b5"],"layout":"IPY_MODEL_1a8b9ac751aa4ed7ac3738dc8ef192b1"}},"69c89cf5dc554b6ea4ede63cc03250a0":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"7558649d528343aa8add50f9df7936b7":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"100%","description_tooltip":null,"layout":"IPY_MODEL_5ac8b2edfc19497e8399eaee41c3d674","max":32,"min":0,"orientation":"horizontal","style":"IPY_MODEL_c5ebf9008f9e497ebb19cac2d37a8c85","value":32}},"76083bf6a73b453fa990e4afef82a7d0":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"7645fbc81ad748deb1850b4124c5ee31":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_17ffee35f43b4c86be709e5512b1637d","placeholder":"​","style":"IPY_MODEL_7c50f32b6cf143fda74846e42fffb580","value":" 1954/1954 [02:47&lt;00:00, 11.66it/s]"}},"7c2120e5f84948aa91a426afe1cd165c":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"7c50f32b6cf143fda74846e42fffb580":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"7d607f9e4e5f4b099c21827d64ff7ee4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"7d8596c77dc84ee18860aad7f615735e":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"7d9d60286c9c4423a914f82b70887e6e":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"835507337a1545da8edaf48ec8aec304":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"100%","description_tooltip":null,"layout":"IPY_MODEL_f509a7ab726a48f6ae9c69d7836a2199","max":250,"min":0,"orientation":"horizontal","style":"IPY_MODEL_3feb1011761142f39f245d4e5e5b367d","value":250}},"845faa14189e4b86992988142f9a88cf":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"Downloading: 100%","description_tooltip":null,"layout":"IPY_MODEL_06dba3f6f8904669bff76540450741c1","max":871891,"min":0,"orientation":"horizontal","style":"IPY_MODEL_f16a5fe8db6846638519765ba4c3398f","value":871891}},"8591de5d48e44ed5953657c582b84015":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_835507337a1545da8edaf48ec8aec304","IPY_MODEL_125be473610c43c08dde4e8139e504a4"],"layout":"IPY_MODEL_4d0025e1d8b44b7d92741efbc82b998f"}},"8ab38fea4f2d42218a824dcfe29dd882":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_845faa14189e4b86992988142f9a88cf","IPY_MODEL_bc60387884b743d7b30954bb9e77c620"],"layout":"IPY_MODEL_c9343957fb694977b2076ac342057180"}},"999141df35af494e8920c6308ce59985":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"9cce84f32aa1449c8759c315729ca637":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_575f00f44a2b41949bcad32a90946707","IPY_MODEL_7645fbc81ad748deb1850b4124c5ee31"],"layout":"IPY_MODEL_7c2120e5f84948aa91a426afe1cd165c"}},"a6e2ac8a7ebe4ac6ae1f3bc5a90dd6aa":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"b330f9456d5c43328e36521e0c239002":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"b595f2bf32974bfda9164f3e59359b5a":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"Downloading: 100%","description_tooltip":null,"layout":"IPY_MODEL_999141df35af494e8920c6308ce59985","max":999358484,"min":0,"orientation":"horizontal","style":"IPY_MODEL_2f89af4eb67540e889c1f0ad0992eaed","value":999358484}},"bc60387884b743d7b30954bb9e77c620":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_f68af62860c94396b62eb403b6bf56c4","placeholder":"​","style":"IPY_MODEL_0265fa838c634a849c25f9b1762500dc","value":" 872k/872k [00:00&lt;00:00, 1.46MB/s]"}},"c5ebf9008f9e497ebb19cac2d37a8c85":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"c9343957fb694977b2076ac342057180":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"ca6213b819c34c22816b292c2b75720c":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_7d9d60286c9c4423a914f82b70887e6e","placeholder":"​","style":"IPY_MODEL_7d607f9e4e5f4b099c21827d64ff7ee4","value":" 569/569 [00:00&lt;00:00, 601B/s]"}},"d800cd3318b64aae87c99b3e3fe26fa3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"e10014a15736446cbb489b47d0712f7b":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"e174a5e3402744dbbd4385d9ceccc4b5":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_503897d0e3c54e8a90a0c2c2b013983a","placeholder":"​","style":"IPY_MODEL_69c89cf5dc554b6ea4ede63cc03250a0","value":" 999M/999M [00:34&lt;00:00, 29.1MB/s]"}},"e32dbfb98ad946c3aa20049cc4aaaf38":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"e881aaefb31843d0857eb62a7066c9d2":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f16a5fe8db6846638519765ba4c3398f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"f509a7ab726a48f6ae9c69d7836a2199":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f68af62860c94396b62eb403b6bf56c4":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"fec722f174f74b278a8f5de9de920365":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_249b02767e1d4fc982eb8b009170ea2d","IPY_MODEL_ca6213b819c34c22816b292c2b75720c"],"layout":"IPY_MODEL_b330f9456d5c43328e36521e0c239002"}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":4}