I created a multilingual train dataset using translates with Google API. And suggest sharing other variations of training dataset on this topic.
I created a multilingual train dataset using translates with Google API. And suggest sharing other variations of training dataset on this topic.
Hey Michael! Amazing work!\nThanks for sharing
", "rawMarkdown": "Hey Michael! Amazing work!\nThanks for sharing", "votes": null }, { "id": "798863", "postDate": "04/05/2020 23:40:23", "content": "I've only eyeballed the data but it looks like some of the translated comments aren't aligned with their IDs?
", "rawMarkdown": "I've only eyeballed the data but it looks like some of the translated comments aren't aligned with their IDs?", "votes": null }, { "id": "799202", "postDate": "04/06/2020 09:10:02", "content": "Nice job, thanks for sharing @miklgr500! I am wondering which library did you use, translate-python or googletrans?
", "rawMarkdown": "Nice job, thanks for sharing @miklgr500! I am wondering which library did you use, [translate-python ](https://github.com/terryyin/translate-python/) or [googletrans](https://github.com/ssut/py-googletrans)?", "votes": null }, { "id": "799280", "postDate": "04/06/2020 10:35:06", "content": "Yep, thank you. I need 3 hours to fix it.
", "rawMarkdown": "Yep, thank you. I need 3 hours to fix it.", "votes": null }, { "id": "799436", "postDate": "04/06/2020 12:52:08", "content": "Hi Michael, a huge thanks for your effort during this competition .\nPlease did u fix the ID of comments and i wonder how much this dataset improve your score ?
", "rawMarkdown": "Hi Michael, a huge thanks for your effort during this competition .\nPlease did u fix the ID of comments and i wonder how much this dataset improve your score ?", "votes": null }, { "id": "799503", "postDate": "04/06/2020 13:49:36", "content": "@haythemtellili5 @leecming the new version of the dataset is available.
", "rawMarkdown": "haythemtellili5 @leecming the new version of the dataset is available.", "votes": null }, { "id": "799508", "postDate": "04/06/2020 13:53:12", "content": "Wow, how did you overcome api limits? Awesome work
", "rawMarkdown": "Wow, how did you overcome api limits? Awesome work", "votes": null }, { "id": "799511", "postDate": "04/06/2020 13:54:51", "content": "I used translators library with some modification for accelerating translation and multiprocessing.
", "rawMarkdown": "I used [translators](https://pypi.org/project/translators/) library with some modification for accelerating translation and [multiprocessing](https://docs.python.org/3/library/multiprocessing.html).", "votes": null }, { "id": "799531", "postDate": "04/06/2020 14:20:26", "content": "Thank, it'll be interesting to see how this shakes things up!
", "rawMarkdown": "Thank, it'll be interesting to see how this shakes things up!", "votes": null }, { "id": "799552", "postDate": "04/06/2020 14:33:33", "content": "It feature of thetranslators library.
And which API did you use there?
", "rawMarkdown": "And which API did you use there?", "votes": null }, { "id": "799642", "postDate": "04/06/2020 15:52:53", "content": "Nice job! Upvoting the post and the dataset as well.
", "rawMarkdown": "Nice job! Upvoting the post and the dataset as well.", "votes": null }, { "id": "799894", "postDate": "04/06/2020 21:41:58", "content": "Thanks for sharing.\nWhat is the difference between the cleaned and not clean version ?
", "rawMarkdown": "Thanks for sharing.\nWhat is the difference between the cleaned and not clean version ?", "votes": null }, { "id": "799970", "postDate": "04/07/2020 00:05:17", "content": "I tried translators for translate but I cannot translate all.(it does not include sleep).\nI used .google method.
\n\nDid you use sleep or other waiting function?
", "rawMarkdown": "I tried translators for translate but I cannot translate all.(it does not include sleep).\nI used .google method.\n\nDid you use sleep or other waiting function?", "votes": null }, { "id": "800097", "postDate": "04/07/2020 04:49:35", "content": "How to use \"translators\" for comments translation
", "rawMarkdown": "[How to use \"translators\" for comments translation ](https://www.kaggle.com/miklgr500/how-to-use-translators-for-comments-translation/log?scriptVersionId=31557415)", "votes": null }, { "id": "800524", "postDate": "04/07/2020 13:40:08", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!", "votes": null }, { "id": "800536", "postDate": "04/07/2020 13:54:39", "content": "Cleaned files dosn't have NaN values.
", "rawMarkdown": "Cleaned files dosn't have NaN values.", "votes": null }, { "id": "804845", "postDate": "04/12/2020 03:11:51", "content": "Would you mind sharing the modified function? Would make it easier to share results.
", "rawMarkdown": "Would you mind sharing the modified function? Would make it easier to share results.", "votes": null }, { "id": "806710", "postDate": "04/14/2020 01:23:17", "content": "Thanks for sharing! Wondering if training on this translated dataset helped? I tried training mBert on the concatenated version of translated version but I only get a score of 0.868 on the test set.
@aroraaman There is a obvious difference between translated text and natural text, but it helped for me.
", "rawMarkdown": "aroraaman There is a obvious difference between translated text and natural text, but it helped for me.", "votes": null }, { "id": "806811", "postDate": "04/14/2020 05:06:12", "content": "It’s not as simple as training a mBert or XLM-R on translated text right? I am having trouble being able to replicate any of the notebook scores that show 0.9389 but when I submit its 0.8582 or something.
\n\nThere are obvious smarter tricks involved I assume?
", "rawMarkdown": "It’s not as simple as training a mBert or XLM-R on translated text right? I am having trouble being able to replicate any of the notebook scores that show 0.9389 but when I submit its 0.8582 or something. \n\nThere are obvious smarter tricks involved I assume?", "votes": null }, { "id": "808598", "postDate": "04/15/2020 14:30:06", "content": "Hey @leecming , did the multilingual training data help you to improve your score ?
", "rawMarkdown": "Hey @leecming , did the multilingual training data help you to improve your score ?", "votes": null }, { "id": "808684", "postDate": "04/15/2020 14:59:00", "content": "@aroraaman You mean you couldn't replicate the result even if you just re-run the public kernel? If so it's a bit strange..
", "rawMarkdown": "aroraaman You mean you couldn't replicate the result even if you just re-run the public kernel? If so it's a bit strange..", "votes": null }, { "id": "811785", "postDate": "04/18/2020 08:23:56", "content": "Great dataset, Michael ! \nAdding this dataset as it is to the training data did not improve my cv/PL scores.\nAccording to @mgornergoogle comment \" ...gut feeling however is that it will not be a very useful approach because automatic translations tend to de-toxify ...\" and also here: \"...all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.\"
\n\nDo you see any ways to translate without losing toxicity in text ? Do you think it is feasible that some of us could build such \"unbiased\" translator in-house ?
\n\nN.B. Just noticed alternative observation: according to @andypenrose here \"...it looks like toxicity does survive the automatic translation process...\"
", "rawMarkdown": "Great dataset, Michael ! \nAdding this dataset as it is to the training data did not improve my cv/PL scores.\nAccording to @mgornergoogle [comment](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/138576#786152) \" ...gut feeling however is that it will not be a very useful approach because automatic translations tend to de-toxify ...\" and also [here:](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/138671#786521) \"...all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.\"\n\nDo you see any ways to translate without losing toxicity in text ? Do you think it is feasible that some of us could build such \"unbiased\" translator in-house ?\n\nN.B. Just noticed alternative observation: according to @andypenrose [here](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/138671#786521) \"...it looks like toxicity does survive the automatic translation process...\"", "votes": null }, { "id": "812341", "postDate": "04/18/2020 17:12:02", "content": "One more dataset, which may be useful for someone.\n[Jigsaw] Multilingual swear profanity
", "rawMarkdown": "One more dataset, which may be useful for someone.\n[[Jigsaw] Multilingual swear profanity](https://www.kaggle.com/miklgr500/jigsaw-multilingual-swear-profanity)", "votes": null }, { "id": "812439", "postDate": "04/18/2020 18:47:26", "content": "@aroraaman : maybe you cannot replicate the result due to the multi-cores TPU training, in that it can give you varying results due to the data sharding. You can refer to https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/140706
", "rawMarkdown": "aroraaman : maybe you cannot replicate the result due to the multi-cores TPU training, in that it can give you varying results due to the data sharding. You can refer to https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/140706", "votes": null }, { "id": "812589", "postDate": "04/18/2020 20:44:18", "content": "I obtain a little better results on my models, but it needs the right multilingual data splitting.\nBut, I must agree with you, that translating a bit de-toxify comments. The real multilingual comments obtain specific words and patterns, which translation distorted. \n Because of it, I start to think about:\n1. Additional features, which can help save some patterns\n2. Scraped the real data from some websites (i.e. youtube...)
", "rawMarkdown": "I obtain a little better results on my models, but it needs the right multilingual data splitting.\nBut, I must agree with you, that translating a bit de-toxify comments. The real multilingual comments obtain specific words and patterns, which translation distorted. \n Because of it, I start to think about:\n1. Additional features, which can help save some patterns\n2. Scraped the real data from some websites (i.e. youtube...)", "votes": null }, { "id": "813250", "postDate": "04/19/2020 14:11:51", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "819223", "postDate": "04/24/2020 12:30:49", "content": "Hi, @miklgr500 thanks for your works and sharing. Do you also translate jigsaw-unintended-bias-train.csv set as well?
yet to try this in comp .. But yeah for personal use ... Very useful
", "rawMarkdown": "yet to try this in comp .. But yeah for personal use ... Very useful", "votes": null } ], "comments": [ { "id": 798852, "author_name": "ma7555", "author_url": "", "post_date": "04/05/2020 23:13:21", "content": "Hey Michael! Amazing work!\nThanks for sharing
", "votes": null, "replies": [] }, { "id": 798863, "author_name": "leecming", "author_url": "", "post_date": "04/05/2020 23:40:23", "content": "I've only eyeballed the data but it looks like some of the translated comments aren't aligned with their IDs?
", "votes": null, "replies": [ { "id": 799280, "author_name": "miklgr500", "author_url": "", "post_date": "04/06/2020 10:35:06", "content": "Yep, thank you. I need 3 hours to fix it.
", "votes": null, "replies": [] } ] }, { "id": 799202, "author_name": "rafiko1", "author_url": "", "post_date": "04/06/2020 09:10:02", "content": "Nice job, thanks for sharing @miklgr500! I am wondering which library did you use, translate-python or googletrans?
", "votes": null, "replies": [ { "id": 799511, "author_name": "miklgr500", "author_url": "", "post_date": "04/06/2020 13:54:51", "content": "I used translators library with some modification for accelerating translation and multiprocessing.
", "votes": null, "replies": [] }, { "id": 804845, "author_name": "aroraaman", "author_url": "", "post_date": "04/12/2020 03:11:51", "content": "Would you mind sharing the modified function? Would make it easier to share results.
", "votes": null, "replies": [] } ] }, { "id": 799436, "author_name": "haythemtellili5", "author_url": "", "post_date": "04/06/2020 12:52:08", "content": "Hi Michael, a huge thanks for your effort during this competition .\nPlease did u fix the ID of comments and i wonder how much this dataset improve your score ?
", "votes": null, "replies": [ { "id": 799503, "author_name": "miklgr500", "author_url": "", "post_date": "04/06/2020 13:49:36", "content": "@haythemtellili5 @leecming the new version of the dataset is available.
", "votes": null, "replies": [] }, { "id": 799531, "author_name": "leecming", "author_url": "", "post_date": "04/06/2020 14:20:26", "content": "Thank, it'll be interesting to see how this shakes things up!
", "votes": null, "replies": [] }, { "id": 808598, "author_name": "haythemtellili5", "author_url": "", "post_date": "04/15/2020 14:30:06", "content": "Hey @leecming , did the multilingual training data help you to improve your score ?
", "votes": null, "replies": [] } ] }, { "id": 799508, "author_name": "philippsinger", "author_url": "", "post_date": "04/06/2020 13:53:12", "content": "Wow, how did you overcome api limits? Awesome work
", "votes": null, "replies": [ { "id": 799552, "author_name": "miklgr500", "author_url": "", "post_date": "04/06/2020 14:33:33", "content": "It feature of thetranslators library.
And which API did you use there?
", "votes": null, "replies": [] }, { "id": 799970, "author_name": "tereka", "author_url": "", "post_date": "04/07/2020 00:05:17", "content": "I tried translators for translate but I cannot translate all.(it does not include sleep).\nI used .google method.
\n\nDid you use sleep or other waiting function?
", "votes": null, "replies": [] }, { "id": 800097, "author_name": "miklgr500", "author_url": "", "post_date": "04/07/2020 04:49:35", "content": "How to use \"translators\" for comments translation
", "votes": null, "replies": [] }, { "id": 800524, "author_name": "tereka", "author_url": "", "post_date": "04/07/2020 13:40:08", "content": "Thank you for sharing!
", "votes": null, "replies": [] } ] }, { "id": 799642, "author_name": "sachinprabhu", "author_url": "", "post_date": "04/06/2020 15:52:53", "content": "Nice job! Upvoting the post and the dataset as well.
", "votes": null, "replies": [] }, { "id": 799894, "author_name": "ludovick", "author_url": "", "post_date": "04/06/2020 21:41:58", "content": "Thanks for sharing.\nWhat is the difference between the cleaned and not clean version ?
", "votes": null, "replies": [ { "id": 800536, "author_name": "miklgr500", "author_url": "", "post_date": "04/07/2020 13:54:39", "content": "Cleaned files dosn't have NaN values.
", "votes": null, "replies": [] } ] }, { "id": 806710, "author_name": "aroraaman", "author_url": "", "post_date": "04/14/2020 01:23:17", "content": "Thanks for sharing! Wondering if training on this translated dataset helped? I tried training mBert on the concatenated version of translated version but I only get a score of 0.868 on the test set.
@aroraaman There is a obvious difference between translated text and natural text, but it helped for me.
", "votes": null, "replies": [] }, { "id": 806811, "author_name": "aroraaman", "author_url": "", "post_date": "04/14/2020 05:06:12", "content": "It’s not as simple as training a mBert or XLM-R on translated text right? I am having trouble being able to replicate any of the notebook scores that show 0.9389 but when I submit its 0.8582 or something.
\n\nThere are obvious smarter tricks involved I assume?
", "votes": null, "replies": [] }, { "id": 808684, "author_name": "bamps53", "author_url": "", "post_date": "04/15/2020 14:59:00", "content": "@aroraaman You mean you couldn't replicate the result even if you just re-run the public kernel? If so it's a bit strange..
", "votes": null, "replies": [] }, { "id": 812439, "author_name": "glorian", "author_url": "", "post_date": "04/18/2020 18:47:26", "content": "@aroraaman : maybe you cannot replicate the result due to the multi-cores TPU training, in that it can give you varying results due to the data sharding. You can refer to https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/140706
", "votes": null, "replies": [] } ] }, { "id": 811785, "author_name": "isakev", "author_url": "", "post_date": "04/18/2020 08:23:56", "content": "Great dataset, Michael ! \nAdding this dataset as it is to the training data did not improve my cv/PL scores.\nAccording to @mgornergoogle comment \" ...gut feeling however is that it will not be a very useful approach because automatic translations tend to de-toxify ...\" and also here: \"...all public translation engines are biased against strong toxic language. It would be bad if Google translate was outputting profanity-laden sentences without a very good reason.\"
\n\nDo you see any ways to translate without losing toxicity in text ? Do you think it is feasible that some of us could build such \"unbiased\" translator in-house ?
\n\nN.B. Just noticed alternative observation: according to @andypenrose here \"...it looks like toxicity does survive the automatic translation process...\"
", "votes": null, "replies": [ { "id": 812589, "author_name": "miklgr500", "author_url": "", "post_date": "04/18/2020 20:44:18", "content": "I obtain a little better results on my models, but it needs the right multilingual data splitting.\nBut, I must agree with you, that translating a bit de-toxify comments. The real multilingual comments obtain specific words and patterns, which translation distorted. \n Because of it, I start to think about:\n1. Additional features, which can help save some patterns\n2. Scraped the real data from some websites (i.e. youtube...)
", "votes": null, "replies": [] } ] }, { "id": 812341, "author_name": "miklgr500", "author_url": "", "post_date": "04/18/2020 17:12:02", "content": "One more dataset, which may be useful for someone.\n[Jigsaw] Multilingual swear profanity
", "votes": null, "replies": [ { "id": 843703, "author_name": "phoenix9032", "author_url": "", "post_date": "05/12/2020 08:00:07", "content": "yet to try this in comp .. But yeah for personal use ... Very useful
", "votes": null, "replies": [] } ] }, { "id": 813250, "author_name": "shahules", "author_url": "", "post_date": "04/19/2020 14:11:51", "content": "Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 819223, "author_name": "ipythonx", "author_url": "", "post_date": "04/24/2020 12:30:49", "content": "Hi, @miklgr500 thanks for your works and sharing. Do you also translate jigsaw-unintended-bias-train.csv set as well?