![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1533864%2F81d6249dfbaba0fcb0da05c7109f855c%2Fmlm-preview.png?generation=1598043639234956&alt=media\")
Hi,
\nSome participants in this competition might have already seen Dezső Ribli's useful notebook Finetune XLM-Roberta on Jigsaw test data with MLM.
\nRecently, I published a notebook Masked, My Dear Watson - MLM with TPU in a getting started competition Contradictory, My Dear Watson. In this notebook, I implement token masking in pure TensorFlow operations, which is used for training models like Bert and (XLM-)Roberta.
\nIf you want to skip all the code irrelevant to this Jigsaw Multilingual Toxic Comment Classification competition, the main implementation is here.
\nThe implemention is a translation into TensorFlow from Hugging Face's code, which is originally in PyTorch.
\nHere are some features in this implementation:
\nThe differences to Dezső Ribli's notebook Finetune XLM-Roberta on Jigsaw test data with MLM are:
\nA visualization of token masking:
\n
I use token masking to further fine tune the loaded model on the competition datasets and external datasets (MNLI + XNLI) with MLM loss as objective, before training the NLI classifier on the labeled dataset.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1533864%2F7b075b70bebfdfa487ce840322df01b8%2Fpretrain%200-vs-pretrain%203.png?generation=1598043493428883&alt=media\")
Here are the observations:
\nThe training is written in a customized way. I would probably try to convert it to model.fit style later, but not sure for now.
Despite the notebook is written for another competition, I still hope you find my notebook helpful and enjoy it.
", "messages": [ { "id": 981898, "postDate": "2020-08-22T20:08:46.580Z", "content": "Hi,
\nSome participants in this competition might have already seen Dezső Ribli's useful notebook Finetune XLM-Roberta on Jigsaw test data with MLM.
\nRecently, I published a notebook Masked, My Dear Watson - MLM with TPU in a getting started competition Contradictory, My Dear Watson. In this notebook, I implement token masking in pure TensorFlow operations, which is used for training models like Bert and (XLM-)Roberta.
\nIf you want to skip all the code irrelevant to this Jigsaw Multilingual Toxic Comment Classification competition, the main implementation is here.
\nThe implemention is a translation into TensorFlow from Hugging Face's code, which is originally in PyTorch.
\nHere are some features in this implementation:
\nThe differences to Dezső Ribli's notebook Finetune XLM-Roberta on Jigsaw test data with MLM are:
\nA visualization of token masking:
\n
I use token masking to further fine tune the loaded model on the competition datasets and external datasets (MNLI + XNLI) with MLM loss as objective, before training the NLI classifier on the labeled dataset.
\n
Here are the observations:
\nThe training is written in a customized way. I would probably try to convert it to model.fit style later, but not sure for now.
Despite the notebook is written for another competition, I still hope you find my notebook helpful and enjoy it.
", "rawMarkdown": "Hi,\n\nSome participants in this competition might have already seen [Dezső Ribli](https://www.kaggle.com/riblidezso)'s useful notebook [Finetune XLM-Roberta on Jigsaw test data with MLM](https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm).\n\nRecently, I published a notebook [Masked, My Dear Watson - MLM with TPU](https://www.kaggle.com/yihdarshieh/masked-my-dear-watson-mlm-with-tpu) in a getting started competition [Contradictory, My Dear Watson](https://www.kaggle.com/c/contradictory-my-dear-watson). In this notebook, I implement token masking in pure [TensorFlow](https://www.tensorflow.org/) operations, which is used for training models like Bert and (XLM-)Roberta.\n\nIf you want to skip all the code irrelevant to this [Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/overview) competition, the main implementation is [here](https://www.kaggle.com/yihdarshieh/masked-my-dear-watson-mlm-with-tpu#mask-tokens).\n\nThe implemention is a translation into [TensorFlow](https://www.tensorflow.org/) from [Hugging Face](https://github.com/huggingface/transformers/blob/390c1285925dd119705e69a266202ef04490d012/examples/distillation/distiller.py)'s code, which is originally in [PyTorch](https://pytorch.org/).\n\nHere are some features in this implementation:\n - dynamic token masking in pure TensorFlow operations, therefore it could be used as a tf.data.Dataset transformation.\n - the number of tokens to mask is calculated based on non-padding tokens (and optionally, excluding other special tokens)\n - including a smoothing option to mask rare tokens more frequently\n\nThe differences to [Dezső Ribli](https://www.kaggle.com/riblidezso)'s notebook [Finetune XLM-Roberta on Jigsaw test data with MLM](https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm) are:\n\n - [Dezső Ribli](https://www.kaggle.com/riblidezso)'s notebook implements token masking in numpy operations, and the tokens are statically masked before MLM finetuning. In this notebook, the maksing is implemented in pure TensorFlow operations, and the tokens are masked dynamically during MLM finetuning as tf.data.Dataset transformation.\n - In [Dezső Ribli](https://www.kaggle.com/riblidezso)'s notebook, the number of tokens to be maksed is calculated before ignoring special tokens (in particular, the [PAD] token). However, in the case where we have a lot of padding tokens due to short texts, we will get a higher ratio of token being masked. This notebook carefully determines the number of tokens to be maksed based on the number the actual (non-special) tokens.\n\nA visualization of token masking:\n\n\n\nI use token masking to further fine tune the loaded model on the competition datasets and external datasets (MNLI + XNLI) with MLM loss as objective, before training the NLI classifier on the labeled dataset.\n\n\n\nHere are the observations:\n\n - After a few epochs (usually 3-6 epochs), the classifier using a (further) pretrained LM model outperforms the one without using further pretrained LM model by 1 ~ 2 %.\n - At the first few epochs, further pretraining gives worse results. A possible explanation is that the further pretraining makes a LM model lose some of its overall language knowledge, therefore in an early stage of training on labeled datasets, it can't extract good features for predicting labels. However, once the training epochs increases, the knowledge gained from further pretraining can help it to better predict a downstream task labels.\n - However, in some different notebook runnings, I saw the same performance for training with/without using further pretraining. Probably the training/validation split and the sampling from external datasets (for pretraining) also play some role here.\n\nThe training is written in a customized way. I would probably try to convert it to `model.fit` style later, but not sure for now.\n\nDespite the notebook is written for another competition, I still hope you find my notebook helpful and enjoy it.", "votes": 2 } ], "comments": [], "raw_markdown_by_id": { "981898": "Hi,\n\nSome participants in this competition might have already seen [Dezső Ribli](https://www.kaggle.com/riblidezso)'s useful notebook [Finetune XLM-Roberta on Jigsaw test data with MLM](https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm).\n\nRecently, I published a notebook [Masked, My Dear Watson - MLM with TPU](https://www.kaggle.com/yihdarshieh/masked-my-dear-watson-mlm-with-tpu) in a getting started competition [Contradictory, My Dear Watson](https://www.kaggle.com/c/contradictory-my-dear-watson). In this notebook, I implement token masking in pure [TensorFlow](https://www.tensorflow.org/) operations, which is used for training models like Bert and (XLM-)Roberta.\n\nIf you want to skip all the code irrelevant to this [Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/overview) competition, the main implementation is [here](https://www.kaggle.com/yihdarshieh/masked-my-dear-watson-mlm-with-tpu#mask-tokens).\n\nThe implemention is a translation into [TensorFlow](https://www.tensorflow.org/) from [Hugging Face](https://github.com/huggingface/transformers/blob/390c1285925dd119705e69a266202ef04490d012/examples/distillation/distiller.py)'s code, which is originally in [PyTorch](https://pytorch.org/).\n\nHere are some features in this implementation:\n - dynamic token masking in pure TensorFlow operations, therefore it could be used as a tf.data.Dataset transformation.\n - the number of tokens to mask is calculated based on non-padding tokens (and optionally, excluding other special tokens)\n - including a smoothing option to mask rare tokens more frequently\n\nThe differences to [Dezső Ribli](https://www.kaggle.com/riblidezso)'s notebook [Finetune XLM-Roberta on Jigsaw test data with MLM](https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm) are:\n\n - [Dezső Ribli](https://www.kaggle.com/riblidezso)'s notebook implements token masking in numpy operations, and the tokens are statically masked before MLM finetuning. In this notebook, the maksing is implemented in pure TensorFlow operations, and the tokens are masked dynamically during MLM finetuning as tf.data.Dataset transformation.\n - In [Dezső Ribli](https://www.kaggle.com/riblidezso)'s notebook, the number of tokens to be maksed is calculated before ignoring special tokens (in particular, the [PAD] token). However, in the case where we have a lot of padding tokens due to short texts, we will get a higher ratio of token being masked. This notebook carefully determines the number of tokens to be maksed based on the number the actual (non-special) tokens.\n\nA visualization of token masking:\n\n\n\nI use token masking to further fine tune the loaded model on the competition datasets and external datasets (MNLI + XNLI) with MLM loss as objective, before training the NLI classifier on the labeled dataset.\n\n\n\nHere are the observations:\n\n - After a few epochs (usually 3-6 epochs), the classifier using a (further) pretrained LM model outperforms the one without using further pretrained LM model by 1 ~ 2 %.\n - At the first few epochs, further pretraining gives worse results. A possible explanation is that the further pretraining makes a LM model lose some of its overall language knowledge, therefore in an early stage of training on labeled datasets, it can't extract good features for predicting labels. However, once the training epochs increases, the knowledge gained from further pretraining can help it to better predict a downstream task labels.\n - However, in some different notebook runnings, I saw the same performance for training with/without using further pretraining. Probably the training/validation split and the sampling from external datasets (for pretraining) also play some role here.\n\nThe training is written in a customized way. I would probably try to convert it to `model.fit` style later, but not sure for now.\n\nDespite the notebook is written for another competition, I still hope you find my notebook helpful and enjoy it." } }