Now it's officially allowed to use translated test set.(see this post)
\n\nEven though test set is now translated to English, it looks still different from original English text. To evaluate your model performance, it makes more sense if you use translated validation set which is translated by same way as test set.(google translate api)\nSo, I also created translated validation set and notebook as a example.\nPlease check it out!\n- translated valid notebook\n- translated valid set\n- translated test set
", "messages": [ { "id": "786440", "postDate": "03/25/2020 22:27:55", "content": "Now it's officially allowed to use translated test set.(see this post)
\n\nEven though test set is now translated to English, it looks still different from original English text. To evaluate your model performance, it makes more sense if you use translated validation set which is translated by same way as test set.(google translate api)\nSo, I also created translated validation set and notebook as a example.\nPlease check it out!\n- translated valid notebook\n- translated valid set\n- translated test set
", "rawMarkdown": "Now it's officially allowed to use translated test set.(see [this post](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/138576#786152)) \n\nEven though test set is now translated to English, it looks still different from original English text. To evaluate your model performance, it makes more sense if you use translated validation set which is translated by same way as test set.(google translate api)\nSo, I also created translated validation set and notebook as a example.\nPlease check it out!\n- [translated valid notebook](https://www.kaggle.com/bamps53/pytorch-tpu-training-with-translated-validation?scriptVersionId=30841456)\n- [translated valid set](https://www.kaggle.com/bamps53/val-en-df)\n- [translated test set](https://www.kaggle.com/bamps53/test-en-df)", "votes": null }, { "id": "786521", "postDate": "03/26/2020 00:29:39", "content": "With this validation set based on translations, you might be optimizing your model for the data + the biases of the translation engine. I'm pretty sure all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.
", "rawMarkdown": "With this validation set based on translations, you might be optimizing your model for the data + the biases of the translation engine. I'm pretty sure all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.", "votes": null }, { "id": "786539", "postDate": "03/26/2020 00:59:33", "content": "@mgornergoogle Yes, I know what you said. I'll look for another creative solution other than straight translation. But so far it's better to use translated test set and I guess most of the current top score is using translated test set. So, I just wanted to share the way to measure the performance on translated test set. I hope this won't be the best way to deal with multilingual text classification.
", "rawMarkdown": "mgornergoogle Yes, I know what you said. I'll look for another creative solution other than straight translation. But so far it's better to use translated test set and I guess most of the current top score is using translated test set. So, I just wanted to share the way to measure the performance on translated test set. I hope this won't be the best way to deal with multilingual text classification.", "votes": null }, { "id": "787262", "postDate": "03/26/2020 16:45:51", "content": "That makes sense. Although, experimenting with Abhishek's latest notebook, https://www.kaggle.com/abhishek/inference-of-bert-tpu-model-ml-w-validation, I find that with the same multilingual model, the translated text scores very slightly better on the LB than the original text. So it looks like toxicity does survive the automatic translation process. A quick test with Google Translate also suggests profanity in -> profanity out.
\n\nPS Thanks to @abhishek for sharing the notebook and others for sharing the translated data.
", "rawMarkdown": "That makes sense. Although, experimenting with Abhishek's latest notebook, https://www.kaggle.com/abhishek/inference-of-bert-tpu-model-ml-w-validation, I find that with the same multilingual model, the translated text scores very slightly better on the LB than the original text. So it looks like toxicity does survive the automatic translation process. A quick test with Google Translate also suggests profanity in -> profanity out.\n\nPS Thanks to @abhishek for sharing the notebook and others for sharing the translated data.", "votes": null }, { "id": "787585", "postDate": "03/26/2020 22:39:17", "content": "Thanks for sharing! @bamps53
", "rawMarkdown": "Thanks for sharing! @bamps53", "votes": null } ], "comments": [ { "id": 786521, "author_name": "mgorner", "author_url": "", "post_date": "03/26/2020 00:29:39", "content": "With this validation set based on translations, you might be optimizing your model for the data + the biases of the translation engine. I'm pretty sure all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.
", "votes": null, "replies": [ { "id": 786539, "author_name": "bamps53", "author_url": "", "post_date": "03/26/2020 00:59:33", "content": "@mgornergoogle Yes, I know what you said. I'll look for another creative solution other than straight translation. But so far it's better to use translated test set and I guess most of the current top score is using translated test set. So, I just wanted to share the way to measure the performance on translated test set. I hope this won't be the best way to deal with multilingual text classification.
", "votes": null, "replies": [] }, { "id": 787262, "author_name": "andypenrose", "author_url": "", "post_date": "03/26/2020 16:45:51", "content": "That makes sense. Although, experimenting with Abhishek's latest notebook, https://www.kaggle.com/abhishek/inference-of-bert-tpu-model-ml-w-validation, I find that with the same multilingual model, the translated text scores very slightly better on the LB than the original text. So it looks like toxicity does survive the automatic translation process. A quick test with Google Translate also suggests profanity in -> profanity out.
\n\nPS Thanks to @abhishek for sharing the notebook and others for sharing the translated data.
", "votes": null, "replies": [] } ] }, { "id": 787585, "author_name": "hamditarek", "author_url": "", "post_date": "03/26/2020 22:39:17", "content": "Thanks for sharing! @bamps53
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "786440": "Now it's officially allowed to use translated test set.(see [this post](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/138576#786152)) \n\nEven though test set is now translated to English, it looks still different from original English text. To evaluate your model performance, it makes more sense if you use translated validation set which is translated by same way as test set.(google translate api)\nSo, I also created translated validation set and notebook as a example.\nPlease check it out!\n- [translated valid notebook](https://www.kaggle.com/bamps53/pytorch-tpu-training-with-translated-validation?scriptVersionId=30841456)\n- [translated valid set](https://www.kaggle.com/bamps53/val-en-df)\n- [translated test set](https://www.kaggle.com/bamps53/test-en-df)", "786521": "With this validation set based on translations, you might be optimizing your model for the data + the biases of the translation engine. I'm pretty sure all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.", "786539": "mgornergoogle Yes, I know what you said. I'll look for another creative solution other than straight translation. But so far it's better to use translated test set and I guess most of the current top score is using translated test set. So, I just wanted to share the way to measure the performance on translated test set. I hope this won't be the best way to deal with multilingual text classification.", "787262": "That makes sense. Although, experimenting with Abhishek's latest notebook, https://www.kaggle.com/abhishek/inference-of-bert-tpu-model-ml-w-validation, I find that with the same multilingual model, the translated text scores very slightly better on the LB than the original text. So it looks like toxicity does survive the automatic translation process. A quick test with Google Translate also suggests profanity in -> profanity out.\n\nPS Thanks to @abhishek for sharing the notebook and others for sharing the translated data.", "787585": "Thanks for sharing! @bamps53" }, "source": "meta" }