{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🦄 Overview\nThis is a improved version of [rafiko1's 1st place baseline](https://www.kaggle.com/rafiko1/1st-place-baseline-xlm-r-es-it-tr), current version=2.\n\nVersion=1 partially modified below compared to [rafiko1's 1st place baseline](https://www.kaggle.com/rafiko1/1st-place-baseline-xlm-r-es-it-tr):\n- MODEL = 'dccuchile/bert-base-spanish-wwm-uncased'\n- BERT_MODEL = True\n- REPEAT_PL = 6\n\nVersion=1 use monolingual bert models with spanish, you can get below score when submitting version=1 output file \"test-es-dccuchile-bert-base-spanish-wwm-uncased.csv\":\n- public score: 0.9483, private 0.9466 for stage1 best prediction \n- public score: 0.9487, private 0.9469 for stage1 avg prediction \n- public score: 0.9485, private 0.9465 for stage2 avg prediction \n- BTW, the initial submission score is: public 0.95, private 0.9485\n\nSo, the pseudo labels training don't improve the initial submission due to the forgetting nature of Stage2, **I fix this problem in version=2**.\n\nVersion=2 (current version) transfer pseudo labels test data from stage1 to stage2 **compared to version=1**, this can avoid the forgetting nature of Stage2, you can get below score when submitting version=2 (current version) output file \"test-es-dccuchile-bert-base-spanish-wwm-uncased-2.csv\":\n- public score: 0.9491, private 0.9475 for stage1 best prediction\n- public score: 0.9494, private 0.9479 for stage1 avg prediction\n- public score: 0.9505, private 0.9492 for stage2 avg prediction\n- BTW, the initial submission score is: public 0.95, private 0.9485\n\nWe can find this change can make pseudo labels strategy works.","metadata":{}},{"cell_type":"markdown","source":"# 🎨 Monolingual Bert Models\nI sorted out the following Monolingual Bert Models from the transformers library\n- [french](https://huggingface.co/camembert/camembert-large)\n- [russian](https://huggingface.co/blinoff/roberta-base-russian-v0)\n- [spanish](https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased)\n- [turkish](https://huggingface.co/dbmdz/bert-base-turkish-cased)\n- [italian](https://huggingface.co/dbmdz/bert-base-italian-xxl-uncased)\n- [portuguese](https://huggingface.co/neuralmind/bert-large-portuguese-cased)\n\nIf it helps you, please upvote me!","metadata":{}},{"cell_type":"markdown","source":"# 📚 Imports","metadata":{"papermill":{"duration":0.011438,"end_time":"2020-08-20T02:40:51.004572","exception":false,"start_time":"2020-08-20T02:40:50.993134","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# General imports\nimport os\nimport numpy as np\nimport pandas as pd\nfrom tqdm.notebook import tqdm\nfrom joblib import Parallel, delayed\n\n# Tensorflow\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Dense, Input\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.callbacks import ModelCheckpoint\n\n# Transformers\nfrom transformers import TFAutoModel, TFBertModel, AutoTokenizer","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-08-20T02:40:51.036303Z","iopub.status.busy":"2020-08-20T02:40:51.035493Z","iopub.status.idle":"2020-08-20T02:40:59.346699Z","shell.execute_reply":"2020-08-20T02:40:59.345819Z"},"papermill":{"duration":8.330663,"end_time":"2020-08-20T02:40:59.346839","exception":false,"start_time":"2020-08-20T02:40:51.016176","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TPU and configs","metadata":{"papermill":{"duration":0.011237,"end_time":"2020-08-20T02:40:59.37005","exception":false,"start_time":"2020-08-20T02:40:59.358813","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:40:59.49578Z","iopub.status.busy":"2020-08-20T02:40:59.494896Z","iopub.status.idle":"2020-08-20T02:41:04.119705Z","shell.execute_reply":"2020-08-20T02:41:04.119064Z"},"papermill":{"duration":4.738114,"end_time":"2020-08-20T02:41:04.119843","exception":false,"start_time":"2020-08-20T02:40:59.381729","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Configuration parameters\nAUTO = tf.data.experimental.AUTOTUNE\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nMAX_LEN = 192\nMODEL = 'dccuchile/bert-base-spanish-wwm-uncased' # for BERT model replace by e.g. dccuchile/bert-base-spanish-wwm-uncased   # jplu/tf-xlm-roberta-large\nLANG = \"es\" # can be any of es, it, tr in this notebook\nCONSTANT_LR = 3e-6 # 3e-6 generally good. Set lower e.g. 1e-6 for more finetuning\nBALANCEMENT = [0.8, 0.2] # non-toxic vs. toxic\nBERT_MODEL = True # specify if the given model is a BERT model\nN_EPOCHS = 3 # 3-5 epochs are usually enough. Set higher e.g. 5 for more finetuning\nN_ITER_PER_EPOCH = 10\nPREDICT_START_ITER = 10 # start iteration to predict on test. best iterations found around +-20 (2 full epochs)\n\n# Upgrades\nSTAGE2 = True # resume training with checkpoint of best model\nREPEAT_PL = 6 # Upgrade: repeat PL with train (I repeated 6x on my last subs). Default=0 (no pseudolabels)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:04.152725Z","iopub.status.busy":"2020-08-20T02:41:04.151653Z","iopub.status.idle":"2020-08-20T02:41:04.154843Z","shell.execute_reply":"2020-08-20T02:41:04.154068Z"},"papermill":{"duration":0.02315,"end_time":"2020-08-20T02:41:04.155005","exception":false,"start_time":"2020-08-20T02:41:04.131855","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Functions processing","metadata":{"papermill":{"duration":0.013667,"end_time":"2020-08-20T02:41:04.180998","exception":false,"start_time":"2020-08-20T02:41:04.167331","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def regular_encode(texts, max_len):\n    \"\"\"\n    Tokenizing the texts into their respective IDs using regular batch encoding\n    \n    Accepts: * texts: the text to be tokenize\n             * max_len: max length of text\n    \n    Returns: * array of tokenized IDs \n    \"\"\"\n    enc_di = tokenizer.batch_encode_plus(\n        texts, \n        return_attention_masks=False, \n        return_token_type_ids=False,\n        pad_to_max_length=True,\n        max_length=max_len\n    )\n    \n    return np.array(enc_di['input_ids'])","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:04.213801Z","iopub.status.busy":"2020-08-20T02:41:04.212947Z","iopub.status.idle":"2020-08-20T02:41:04.215764Z","shell.execute_reply":"2020-08-20T02:41:04.216399Z"},"papermill":{"duration":0.023007,"end_time":"2020-08-20T02:41:04.21658","exception":false,"start_time":"2020-08-20T02:41:04.193573","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def parallel_encode(texts, max_len):\n    \"\"\"\n    Tokenizing the texts into their respective IDs using parallel processing\n    \n    Accepts: * texts: the text to be tokenized\n             * max_len: max length of text\n    \n    Returns: * array of tokenized IDs + the toxicity label  \n    \"\"\"\n    enc_di = tokenizer.encode_plus(\n        str(texts[0]),\n        return_attention_masks=False, \n        return_token_type_ids=False,\n        pad_to_max_length=True,\n        max_length=max_len\n    )\n    return np.array(enc_di['input_ids']), texts[1]","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:04.249761Z","iopub.status.busy":"2020-08-20T02:41:04.248953Z","iopub.status.idle":"2020-08-20T02:41:04.252887Z","shell.execute_reply":"2020-08-20T02:41:04.252054Z"},"papermill":{"duration":0.02391,"end_time":"2020-08-20T02:41:04.253017","exception":false,"start_time":"2020-08-20T02:41:04.229107","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model(transformer, max_len):\n    \"\"\"\n    Build the model by using transformer layer and simple CLS token\n    \n    Accepts: * transformer: transformer layer\n             * max_len: max length of text\n    \n    Returns: * model \n    \"\"\"\n    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n    sequence_output = transformer(input_word_ids)[0]\n    cls_token = sequence_output[:, 0, :]\n    out = Dense(1, activation='sigmoid')(cls_token)\n    model = Model(inputs=input_word_ids, outputs=out)    \n    return model","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:04.286513Z","iopub.status.busy":"2020-08-20T02:41:04.28569Z","iopub.status.idle":"2020-08-20T02:41:04.28935Z","shell.execute_reply":"2020-08-20T02:41:04.288719Z"},"papermill":{"duration":0.024217,"end_time":"2020-08-20T02:41:04.289489","exception":false,"start_time":"2020-08-20T02:41:04.265272","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create fast tokenizer","metadata":{"papermill":{"duration":0.012221,"end_time":"2020-08-20T02:41:04.314591","exception":false,"start_time":"2020-08-20T02:41:04.30237","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# First load the real tokenizer\ntokenizer = AutoTokenizer.from_pretrained(MODEL)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:04.345319Z","iopub.status.busy":"2020-08-20T02:41:04.34455Z","iopub.status.idle":"2020-08-20T02:41:06.619868Z","shell.execute_reply":"2020-08-20T02:41:06.619129Z"},"papermill":{"duration":2.29245,"end_time":"2020-08-20T02:41:06.619995","exception":false,"start_time":"2020-08-20T02:41:04.327545","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load text data into memory","metadata":{"papermill":{"duration":0.012997,"end_time":"2020-08-20T02:41:06.646789","exception":false,"start_time":"2020-08-20T02:41:06.633792","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train = pd.read_csv(f\"/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-{LANG}-cleaned.csv\")\nvalid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\ntest = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv\")","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:06.683867Z","iopub.status.busy":"2020-08-20T02:41:06.683091Z","iopub.status.idle":"2020-08-20T02:41:08.826091Z","shell.execute_reply":"2020-08-20T02:41:08.825277Z"},"papermill":{"duration":2.16611,"end_time":"2020-08-20T02:41:08.826246","exception":false,"start_time":"2020-08-20T02:41:06.660136","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train), len(valid), len(test))","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:08.859686Z","iopub.status.busy":"2020-08-20T02:41:08.858914Z","iopub.status.idle":"2020-08-20T02:41:08.863132Z","shell.execute_reply":"2020-08-20T02:41:08.862564Z"},"papermill":{"duration":0.023454,"end_time":"2020-08-20T02:41:08.863309","exception":false,"start_time":"2020-08-20T02:41:08.839855","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# if REPEAT_PL:\n#     sub = pd.read_csv(\"../input/multilingual-toxic-comments-training-data/test9500.csv\") # use one of earlier subs\n#     print(sub.head())\n#     print(len(sub))\n#     sub[\"comment_text\"] = test[\"content\"]\n#     sub = sub.loc[test[\"lang\"]==LANG].reset_index(drop=True)\n#     sub_repeat = pd.concat([sub]*REPEAT_PL, ignore_index=True) # repeat PL multipe times for training\n#     print('\\n', sub_repeat.head())\n#     print(len(sub_repeat))\n#     same_cols = [\"comment_text\", \"toxic\"]\n#     train = pd.concat([train[same_cols], sub_repeat[same_cols]]).sample(frac=1).reset_index(drop=True)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:08.896235Z","iopub.status.busy":"2020-08-20T02:41:08.895107Z","iopub.status.idle":"2020-08-20T02:41:08.898568Z","shell.execute_reply":"2020-08-20T02:41:08.897807Z"},"papermill":{"duration":0.021761,"end_time":"2020-08-20T02:41:08.898695","exception":false,"start_time":"2020-08-20T02:41:08.876934","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get specific validation and test\nvalid = valid.loc[valid[\"lang\"]==LANG].reset_index(drop=True)\ntest = test.loc[test[\"lang\"]==LANG].reset_index(drop=True)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:08.95539Z","iopub.status.busy":"2020-08-20T02:41:08.938307Z","iopub.status.idle":"2020-08-20T02:41:08.970436Z","shell.execute_reply":"2020-08-20T02:41:08.969586Z"},"papermill":{"duration":0.058017,"end_time":"2020-08-20T02:41:08.970571","exception":false,"start_time":"2020-08-20T02:41:08.912554","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tokenize","metadata":{"papermill":{"duration":0.013502,"end_time":"2020-08-20T02:41:08.99806","exception":false,"start_time":"2020-08-20T02:41:08.984558","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time \n# Tokenize train with parallel processing\nrows = zip(train['comment_text'].values.tolist(), train.toxic.values.tolist())\nx_y_train = Parallel(n_jobs=4, backend='multiprocessing')(delayed(parallel_encode)(row, max_len=MAX_LEN) for row in tqdm(rows))","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:41:09.046936Z","iopub.status.busy":"2020-08-20T02:41:09.045827Z","iopub.status.idle":"2020-08-20T02:46:02.400482Z","shell.execute_reply":"2020-08-20T02:46:02.401022Z"},"papermill":{"duration":293.389467,"end_time":"2020-08-20T02:46:02.401249","exception":false,"start_time":"2020-08-20T02:41:09.011782","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(x_y_train))\ntry:\n    print(x_y_train.shape)\nexcept:\n    pass\nprint(x_y_train[0])","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:02.437411Z","iopub.status.busy":"2020-08-20T02:46:02.436371Z","iopub.status.idle":"2020-08-20T02:46:02.442018Z","shell.execute_reply":"2020-08-20T02:46:02.441077Z"},"papermill":{"duration":0.026699,"end_time":"2020-08-20T02:46:02.442224","exception":false,"start_time":"2020-08-20T02:46:02.415525","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train = np.vstack(np.array(x_y_train)[:,0])\n\ny_train = np.array(x_y_train)[:,1]\ny_train = np.asarray(y_train).astype('float32').reshape((-1,1))\nprint(y_train.shape)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:02.74265Z","iopub.status.busy":"2020-08-20T02:46:02.597604Z","iopub.status.idle":"2020-08-20T02:46:03.758478Z","shell.execute_reply":"2020-08-20T02:46:03.757798Z"},"papermill":{"duration":1.301081,"end_time":"2020-08-20T02:46:03.758615","exception":false,"start_time":"2020-08-20T02:46:02.457534","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Tokenize valid regular processing\nx_valid = regular_encode(valid.comment_text.values, max_len=MAX_LEN)\n\ny_valid = valid.toxic.values\ny_valid = np.asarray(y_valid).astype('float32').reshape((-1,1)) ","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:03.816683Z","iopub.status.busy":"2020-08-20T02:46:03.80118Z","iopub.status.idle":"2020-08-20T02:46:10.490196Z","shell.execute_reply":"2020-08-20T02:46:10.489221Z"},"papermill":{"duration":6.716949,"end_time":"2020-08-20T02:46:10.490364","exception":false,"start_time":"2020-08-20T02:46:03.773415","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nx_test = regular_encode(test.content.values, max_len=MAX_LEN)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:10.539087Z","iopub.status.busy":"2020-08-20T02:46:10.533638Z","iopub.status.idle":"2020-08-20T02:46:32.232175Z","shell.execute_reply":"2020-08-20T02:46:32.232762Z"},"papermill":{"duration":21.727409,"end_time":"2020-08-20T02:46:32.232935","exception":false,"start_time":"2020-08-20T02:46:10.505526","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build datasets objects","metadata":{"papermill":{"duration":0.014391,"end_time":"2020-08-20T02:46:32.262052","exception":false,"start_time":"2020-08-20T02:46:32.247661","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Train and valid dataset\ntrain_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_train, y_train))\n    .shuffle(buffer_size=len(x_train), seed = 18)\n    .prefetch(AUTO)\n)\n\nvalid_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices((x_valid, y_valid))\n    .batch(BATCH_SIZE)\n    .cache()\n    .prefetch(AUTO)\n)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:33.240192Z","iopub.status.busy":"2020-08-20T02:46:33.236454Z","iopub.status.idle":"2020-08-20T02:46:33.962416Z","shell.execute_reply":"2020-08-20T02:46:33.961728Z"},"papermill":{"duration":1.685724,"end_time":"2020-08-20T02:46:33.962548","exception":false,"start_time":"2020-08-20T02:46:32.276824","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Balance the train dataset by creating seperate negative and positive datasets. \n# Note: tf.squeeze remove the added dim to labels\n# Example taken from https://www.tensorflow.org/guide/data\n\nnegative_ds = (\n  train_dataset\n    .filter(lambda _, y: tf.squeeze(y)==0)\n    .repeat())\n\npositive_ds = (\n  train_dataset\n    .filter(lambda _, y: tf.squeeze(y)==1)\n    .repeat())\n\nbalanced_ds = tf.data.experimental.sample_from_datasets(\n    [negative_ds, positive_ds], BALANCEMENT).batch(BATCH_SIZE) # Around 80%/20% to be expected for 0/1 labels","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:34.001852Z","iopub.status.busy":"2020-08-20T02:46:34.001056Z","iopub.status.idle":"2020-08-20T02:46:34.065962Z","shell.execute_reply":"2020-08-20T02:46:34.066539Z"},"papermill":{"duration":0.089428,"end_time":"2020-08-20T02:46:34.066715","exception":false,"start_time":"2020-08-20T02:46:33.977287","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# distribute the datset according to the strategy\ntrain_dist_ds = strategy.experimental_distribute_dataset(balanced_ds)\nvalid_dist_ds = strategy.experimental_distribute_dataset(valid_dataset)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:34.104411Z","iopub.status.busy":"2020-08-20T02:46:34.103638Z","iopub.status.idle":"2020-08-20T02:46:34.120195Z","shell.execute_reply":"2020-08-20T02:46:34.119557Z"},"papermill":{"duration":0.038293,"end_time":"2020-08-20T02:46:34.120346","exception":false,"start_time":"2020-08-20T02:46:34.082053","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Functions TF custom training","metadata":{"papermill":{"duration":0.014382,"end_time":"2020-08-20T02:46:34.149753","exception":false,"start_time":"2020-08-20T02:46:34.135371","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Instantiate metrics\nwith strategy.scope():\n    # Accuracy, AUC, loss train\n    train_accuracy = tf.keras.metrics.BinaryAccuracy()\n    train_auc = tf.keras.metrics.AUC()\n    train_loss = tf.keras.metrics.Sum()\n    \n    # Accuracy, AUC, loss valid\n    valid_accuracy = tf.keras.metrics.BinaryAccuracy()\n    valid_auc = tf.keras.metrics.AUC()\n    valid_loss = tf.keras.metrics.Sum()\n    \n    # TP, TN, FN, FP train\n    train_TP = tf.keras.metrics.TruePositives()\n    train_TN = tf.keras.metrics.TrueNegatives()\n    train_FP = tf.keras.metrics.FalsePositives()\n    train_FN = tf.keras.metrics.FalseNegatives()\n    \n    # TP, TN, FN, FP valid\n    valid_TP = tf.keras.metrics.TruePositives()\n    valid_TN = tf.keras.metrics.TrueNegatives()\n    valid_FP = tf.keras.metrics.FalsePositives()\n    valid_FN = tf.keras.metrics.FalseNegatives()\n    \n    # Loss function and optimizer\n    loss_fn = lambda a,b: tf.nn.compute_average_loss(tf.keras.losses.binary_crossentropy(a,b), global_batch_size=BATCH_SIZE)\n    optimizer = tf.keras.optimizers.Adam(learning_rate=CONSTANT_LR)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:34.201525Z","iopub.status.busy":"2020-08-20T02:46:34.193352Z","iopub.status.idle":"2020-08-20T02:46:35.202311Z","shell.execute_reply":"2020-08-20T02:46:35.201262Z"},"papermill":{"duration":1.037784,"end_time":"2020-08-20T02:46:35.202454","exception":false,"start_time":"2020-08-20T02:46:34.16467","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"@tf.function\ndef train_step(tokens, labels):\n    with tf.GradientTape() as tape:\n        probabilities = model(tokens, training=True)\n        loss = loss_fn(labels, probabilities)\n    grads = tape.gradient(loss, model.trainable_variables)\n    optimizer.apply_gradients(zip(grads, model.trainable_variables))\n    \n    # update metrics\n    train_accuracy.update_state(labels, probabilities)\n    train_auc.update_state(labels, probabilities)\n    train_loss.update_state(loss)\n    \n    train_TP.update_state(labels, probabilities)\n    train_TN.update_state(labels, probabilities)\n    train_FP.update_state(labels, probabilities)\n    train_FN.update_state(labels, probabilities)\n    \n@tf.function\ndef valid_step(tokens, labels):\n    probabilities = model(tokens, training=False)\n    loss = loss_fn(labels, probabilities)\n    \n    # update metrics\n    valid_accuracy.update_state(labels, probabilities)\n    valid_auc.update_state(labels, probabilities)\n    valid_loss.update_state(loss)\n    \n    valid_TP.update_state(labels, probabilities)\n    valid_TN.update_state(labels, probabilities)\n    valid_FP.update_state(labels, probabilities)\n    valid_FN.update_state(labels, probabilities)\n    ","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:35.248641Z","iopub.status.busy":"2020-08-20T02:46:35.247657Z","iopub.status.idle":"2020-08-20T02:46:35.250878Z","shell.execute_reply":"2020-08-20T02:46:35.250123Z"},"papermill":{"duration":0.032443,"end_time":"2020-08-20T02:46:35.251001","exception":false,"start_time":"2020-08-20T02:46:35.218558","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ➕ Load model","metadata":{"papermill":{"duration":0.014445,"end_time":"2020-08-20T02:46:35.280514","exception":false,"start_time":"2020-08-20T02:46:35.266069","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nwith strategy.scope():\n    if BERT_MODEL:\n        transformer_layer = TFBertModel.from_pretrained(MODEL, from_pt=True)\n    else:\n        transformer_layer = TFAutoModel.from_pretrained(MODEL)\n    model = build_model(transformer_layer, max_len=MAX_LEN)\nmodel.summary()","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:46:35.318596Z","iopub.status.busy":"2020-08-20T02:46:35.316553Z","iopub.status.idle":"2020-08-20T02:47:10.593918Z","shell.execute_reply":"2020-08-20T02:47:10.592742Z"},"papermill":{"duration":35.298743,"end_time":"2020-08-20T02:47:10.594124","exception":false,"start_time":"2020-08-20T02:46:35.295381","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📉 Custom training loop","metadata":{"papermill":{"duration":0.015655,"end_time":"2020-08-20T02:47:10.626831","exception":false,"start_time":"2020-08-20T02:47:10.611176","status":"completed"},"tags":[]}},{"cell_type":"code","source":"VALIDATION_STEPS = x_valid.shape[0] // BATCH_SIZE\nSTEPS_PER_EPOCH = x_train.shape[0] // (BATCH_SIZE*N_ITER_PER_EPOCH)\nprint(\"Steps per epoch:\", STEPS_PER_EPOCH)\nEPOCHS = N_EPOCHS*N_ITER_PER_EPOCH\n\nbest_auc = 0\nepoch = 0\n\npreds_all = []\nfor step, (tokens, labels) in enumerate(train_dist_ds):\n    # run training step\n    strategy.experimental_run_v2(train_step, args=(tokens, labels))\n    print('=', end='', flush=True)\n    \n    # print metrics training\n    if ((step+1) // STEPS_PER_EPOCH) > epoch:\n        print(\"\\n Epoch:\", epoch)\n        print('|', end='', flush=True)\n        print(\"TP -  TN  -  FP  -  FN\")\n        print(train_TP.result().numpy(), train_TN.result().numpy(), train_FP.result().numpy(), train_FN.result().numpy())\n        print(\"train AUC: \",train_auc.result().numpy())\n        print(\"train loss: \", train_loss.result().numpy() / STEPS_PER_EPOCH)\n        \n        # validation run for es, it, tr and save model\n        for tokens, labels in valid_dist_ds:\n            strategy.experimental_run_v2(valid_step, args=(tokens, labels))\n            print('=', end='', flush=True)\n\n        # compute metrics\n        print(\"\\n\")\n        print(\"TP -  TN  -  FP  -  FN\")\n        print(valid_TP.result().numpy(), valid_TN.result().numpy(), valid_FP.result().numpy(), valid_FN.result().numpy())\n        print(\"val AUC: \", valid_auc.result().numpy())\n        print(\"val loss: \", valid_loss.result().numpy() / VALIDATION_STEPS)\n\n        # Save predictions and weights of model\n        if (valid_auc.result().numpy() > best_auc) & (epoch>=PREDICT_START_ITER):\n            best_auc = valid_auc.result().numpy()\n            print(\"Prediction on test set - snapshot\")\n            preds = model.predict(test_dataset, verbose = 1)\n            preds_all.append(preds)\n            model.save_weights('best_model.h5') # keep track of best model\n        # set up next epoch\n        epoch = (step+1) // STEPS_PER_EPOCH\n\n        train_auc.reset_states()\n        valid_auc.reset_states()\n\n        valid_loss.reset_states()\n        train_loss.reset_states()\n        \n        train_TP.reset_states()\n        train_TN.reset_states()\n        train_FP.reset_states()\n        train_FN.reset_states()\n        \n        valid_TP.reset_states()\n        valid_TN.reset_states()\n        valid_FP.reset_states()\n        valid_FN.reset_states()\n        \n        if epoch >= EPOCHS:\n            break","metadata":{"execution":{"iopub.execute_input":"2020-08-20T02:47:10.684687Z","iopub.status.busy":"2020-08-20T02:47:10.682552Z","iopub.status.idle":"2020-08-20T03:02:52.515363Z","shell.execute_reply":"2020-08-20T03:02:52.514563Z"},"papermill":{"duration":941.872933,"end_time":"2020-08-20T03:02:52.515496","exception":false,"start_time":"2020-08-20T02:47:10.642563","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💾 Save predictions","metadata":{"papermill":{"duration":0.390664,"end_time":"2020-08-20T03:02:53.343524","exception":false,"start_time":"2020-08-20T03:02:52.95286","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#Generate averages of predictions: last one, and average of snapshots\ntest[\"toxic_best\"] = preds_all[-1]\ntest[\"toxic_avg\"] = sum(preds_all)/len(preds_all)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:02:54.162375Z","iopub.status.busy":"2020-08-20T03:02:54.161561Z","iopub.status.idle":"2020-08-20T03:02:54.166443Z","shell.execute_reply":"2020-08-20T03:02:54.165842Z"},"papermill":{"duration":0.432636,"end_time":"2020-08-20T03:02:54.166569","exception":false,"start_time":"2020-08-20T03:02:53.733933","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the predictions\nMODEL_NAME = MODEL.replace(\"/\", \"-\")\ntest.to_csv(f\"test-{LANG}-{MODEL_NAME}-2-1.csv\", index=False)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:02:54.993584Z","iopub.status.busy":"2020-08-20T03:02:54.992503Z","iopub.status.idle":"2020-08-20T03:02:55.223674Z","shell.execute_reply":"2020-08-20T03:02:55.22298Z"},"papermill":{"duration":0.626158,"end_time":"2020-08-20T03:02:55.223812","exception":false,"start_time":"2020-08-20T03:02:54.597654","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ⌨️ Stage 2: resume training on validation data","metadata":{"papermill":{"duration":0.388789,"end_time":"2020-08-20T03:02:56.012299","exception":false,"start_time":"2020-08-20T03:02:55.62351","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Build datasets objects","metadata":{"papermill":{"duration":0.389133,"end_time":"2020-08-20T03:02:56.830621","exception":false,"start_time":"2020-08-20T03:02:56.441488","status":"completed"},"tags":[]}},{"cell_type":"code","source":"valid = pd.read_csv('/kaggle/input/jigsaw-multilingual-toxic-comment-classification/validation.csv')\nvalid = valid.loc[valid[\"lang\"]==LANG].reset_index(drop=True)\ntest = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/test.csv\")\n\nif REPEAT_PL:\n    sub = pd.read_csv(\"../input/multilingual-toxic-comments-training-data/test9500.csv\") # use one of earlier subs\n    print(sub.head())\n    print(len(sub))\n    sub[\"comment_text\"] = test[\"content\"]\n    sub = sub.loc[test[\"lang\"]==LANG].reset_index(drop=True)\n    sub_repeat = pd.concat([sub]*REPEAT_PL, ignore_index=True) # repeat PL multipe times for training\n    print('\\n', sub_repeat.head())\n    print(len(sub_repeat))\n    same_cols = [\"comment_text\", \"toxic\"]\n    valid = pd.concat([valid[same_cols], sub_repeat[same_cols]]).sample(frac=1).reset_index(drop=True)\n\ntest = test.loc[test[\"lang\"]==LANG].reset_index(drop=True)  \n\nx_valid = regular_encode(valid.comment_text.values, max_len=MAX_LEN)\ny_valid = valid.toxic.values\ny_valid = np.asarray(y_valid).astype('float32').reshape((-1,1)) \nx_test = regular_encode(test.content.values, max_len=MAX_LEN)\n\ntest_dataset = (\n    tf.data.Dataset\n    .from_tensor_slices(x_test)\n    .batch(BATCH_SIZE)\n)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:02:57.63081Z","iopub.status.busy":"2020-08-20T03:02:57.630031Z","iopub.status.idle":"2020-08-20T03:05:36.118132Z","shell.execute_reply":"2020-08-20T03:05:36.117396Z"},"papermill":{"duration":158.895876,"end_time":"2020-08-20T03:05:36.118346","exception":false,"start_time":"2020-08-20T03:02:57.22247","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if STAGE2:\n    # the validation set becomes train_dataset\n    train_dataset = (\n        tf.data.Dataset\n        .from_tensor_slices((x_valid, y_valid)) # replaced by x_valid, y_valid!\n        .shuffle(buffer_size=len(x_valid), seed = 18)\n        .prefetch(AUTO)\n        .batch(BATCH_SIZE)\n        .repeat()\n    )\n    \n    # distribute the datset according to the strategy\n    train_dist_ds = strategy.experimental_distribute_dataset(train_dataset)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:05:37.135815Z","iopub.status.busy":"2020-08-20T03:05:37.134687Z","iopub.status.idle":"2020-08-20T03:05:37.321907Z","shell.execute_reply":"2020-08-20T03:05:37.321101Z"},"papermill":{"duration":0.759732,"end_time":"2020-08-20T03:05:37.322032","exception":false,"start_time":"2020-08-20T03:05:36.5623","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📷 Load model","metadata":{"papermill":{"duration":0.441987,"end_time":"2020-08-20T03:05:38.159583","exception":false,"start_time":"2020-08-20T03:05:37.717596","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if STAGE2:\n    model.load_weights(\"best_model.h5\") # best model from stage1","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:05:39.024443Z","iopub.status.busy":"2020-08-20T03:05:39.001369Z","iopub.status.idle":"2020-08-20T03:05:44.286537Z","shell.execute_reply":"2020-08-20T03:05:44.285834Z"},"papermill":{"duration":5.730548,"end_time":"2020-08-20T03:05:44.28667","exception":false,"start_time":"2020-08-20T03:05:38.556122","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Custom training loop","metadata":{"papermill":{"duration":0.440506,"end_time":"2020-08-20T03:05:45.127607","exception":false,"start_time":"2020-08-20T03:05:44.687101","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if STAGE2:\n    STEPS_PER_EPOCH = round(x_valid.shape[0] / (BATCH_SIZE*N_ITER_PER_EPOCH))\n    print(\"Steps per epoch:\", STEPS_PER_EPOCH)\n    EPOCHS = N_EPOCHS*N_ITER_PER_EPOCH\n    best_auc = 0\n    epoch = 0\n\n    preds_all = []\n    for step, (tokens, labels) in enumerate(train_dist_ds):\n        # run training step\n        strategy.experimental_run_v2(train_step, args=(tokens, labels))\n        print('=', end='', flush=True)\n\n        # print metrics training\n        if ((step+1) // STEPS_PER_EPOCH) > epoch:\n            print(\"\\n Epoch:\", epoch)\n            print('|', end='', flush=True)\n            print(\"TP -  TN  -  FP  -  FN\")\n            print(train_TP.result().numpy(), train_TN.result().numpy(), train_FP.result().numpy(), train_FN.result().numpy())\n            print(\"train AUC: \",train_auc.result().numpy())\n            print(\"train loss: \", train_loss.result().numpy() / STEPS_PER_EPOCH)\n\n            # Save predictions and weights of model\n            if epoch>=PREDICT_START_ITER:\n                print(\"Prediction on test set - snapshot\")\n                preds = model.predict(test_dataset, verbose = 1)\n                preds_all.append(preds)\n                \n            # set up next epoch\n            epoch = (step+1) // STEPS_PER_EPOCH\n            \n            train_auc.reset_states()\n            train_loss.reset_states()\n\n            train_TP.reset_states()\n            train_TN.reset_states()\n            train_FP.reset_states()\n            train_FN.reset_states()\n            \n            if epoch >= EPOCHS:\n                # save model if needed\n                model.save_weights('best_model_valid.h5') \n                break","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:05:45.936481Z","iopub.status.busy":"2020-08-20T03:05:45.935364Z","iopub.status.idle":"2020-08-20T03:10:35.376238Z","shell.execute_reply":"2020-08-20T03:10:35.375511Z"},"papermill":{"duration":289.853963,"end_time":"2020-08-20T03:10:35.376375","exception":false,"start_time":"2020-08-20T03:05:45.522412","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 💾 Save predictions","metadata":{"papermill":{"duration":0.56721,"end_time":"2020-08-20T03:10:36.464434","exception":false,"start_time":"2020-08-20T03:10:35.897224","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if STAGE2:\n    #Generate averages of snapshot\n    test[\"toxic_mean_snap_valid\"] = sum(preds_all)/len(preds_all)\n    # Save the predictions\n    MODEL_NAME = MODEL.replace(\"/\", \"-\")\n    test_pre = pd.read_csv(f\"test-{LANG}-{MODEL_NAME}-2-1.csv\")\n    assert sum(test_pre[\"content\"]==test[\"content\"]) == len(test)\n    test[\"toxic_best\"] = test_pre[\"toxic_best\"]\n    test[\"toxic_avg\"] = test_pre[\"toxic_avg\"]\n    \n    test.to_csv(f\"test-{LANG}-{MODEL_NAME}-2.csv\", index=False)","metadata":{"execution":{"iopub.execute_input":"2020-08-20T03:10:37.596359Z","iopub.status.busy":"2020-08-20T03:10:37.595131Z","iopub.status.idle":"2020-08-20T03:10:37.830328Z","shell.execute_reply":"2020-08-20T03:10:37.829511Z"},"papermill":{"duration":0.818782,"end_time":"2020-08-20T03:10:37.830471","exception":false,"start_time":"2020-08-20T03:10:37.011689","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}