![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3406049%2Ff4c10ba9d7f311dcd0d82117e47e2e99%2Fjigsaw_dataset_filtering.png?generation=1590802128615496&alt=media\")
There might be leaked samples from validation data, ~600 samples on test data.\nI could confirm some minor changed validation samples on test data or the exact same samples on test data like below.\nI displayed these samples on the notebook (https://www.kaggle.com/sai11fkaneko/data-leak )
\n\nAnd also I tried whether LB score is improved. But I got worse result if I use this leak.(0.9462 -> 0.9459: base submission file from https://www.kaggle.com/hamditarek/ensemble )\nOne possible reason for this is my rough treatment after 128 tokens. \nBut still not sure for the reason...
\n\n>> TEST \n ID:8667, LANG:tr, Duplicated Num:2\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi . Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.
\n\n>> VALIDATION \n ID:2162, LANG:tr, Duplicated Num:1, TOXIC:0\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi. Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.
\n\n>> diff result\n* TEST\n--- VAL
\n\n* 14,21 **\n alındı ya da! silindi! . Başka bir deneme--- 14,20 ----\n alındı ya da! silindi. Başka bir deneme
\n\n>> TEST ENGLISH Translation\nThank you for doing experiments in Wikipedia. Your experiment worked, but at the moment either rolled back or deleted. If you want to make another attempt, please use the sandbox. If you want to learn more about how you might be contributing to our encyclopedia Take a look at the welcome page. Thank you.
\n\n```
", "messages": [ { "id": "865850", "postDate": "05/28/2020 23:57:33", "content": "There might be leaked samples from validation data, ~600 samples on test data.\nI could confirm some minor changed validation samples on test data or the exact same samples on test data like below.\nI displayed these samples on the notebook (https://www.kaggle.com/sai11fkaneko/data-leak )
\n\nAnd also I tried whether LB score is improved. But I got worse result if I use this leak.(0.9462 -> 0.9459: base submission file from https://www.kaggle.com/hamditarek/ensemble )\nOne possible reason for this is my rough treatment after 128 tokens. \nBut still not sure for the reason...
\n\n>> TEST \n ID:8667, LANG:tr, Duplicated Num:2\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi . Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.
\n\n>> VALIDATION \n ID:2162, LANG:tr, Duplicated Num:1, TOXIC:0\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi. Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.
\n\n>> diff result\n* TEST\n--- VAL
\n\n* 14,21 **\n alındı ya da! silindi! . Başka bir deneme--- 14,20 ----\n alındı ya da! silindi. Başka bir deneme
\n\n>> TEST ENGLISH Translation\nThank you for doing experiments in Wikipedia. Your experiment worked, but at the moment either rolled back or deleted. If you want to make another attempt, please use the sandbox. If you want to learn more about how you might be contributing to our encyclopedia Take a look at the welcome page. Thank you.
\n\n```
", "rawMarkdown": "There might be leaked samples from validation data, ~600 samples on test data.\nI could confirm some minor changed validation samples on test data or the exact same samples on test data like below.\nI displayed these samples on the notebook (https://www.kaggle.com/sai11fkaneko/data-leak )\n\nAnd also I tried whether LB score is improved. But I got worse result if I use this leak.(0.9462 -> 0.9459: base submission file from https://www.kaggle.com/hamditarek/ensemble )\nOne possible reason for this is my rough treatment after 128 tokens. \nBut still not sure for the reason...\n```\n--------------------------------------------------------------------------------\n>> TEST \n ID:8667, LANG:tr, Duplicated Num:2\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi . Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.\n\n>> VALIDATION \n ID:2162, LANG:tr, Duplicated Num:1, TOXIC:0\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi. Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.\n\n>> diff result\n*** TEST\n--- VAL\n***************\n*** 14,21 ****\n alındı ya da! silindi! . Başka bir deneme--- 14,20 ----\n alındı ya da! silindi. Başka bir deneme\n\n>> TEST ENGLISH Translation\nThank you for doing experiments in Wikipedia. Your experiment worked, but at the moment either rolled back or deleted. If you want to make another attempt, please use the sandbox. If you want to learn more about how you might be contributing to our encyclopedia Take a look at the welcome page. Thank you.\n\n```", "votes": null }, { "id": "865902", "postDate": "05/29/2020 01:09:03", "content": "Hi! Thanks for the report. I'll chat with the host and see what they'd like to do about these samples.
\n\nRe: your post - there are some deeply offensive terms in the text, that are not required to make your point. Please remove the last example / English translation from your post, please.
\n\nUpdated to add: thanks for removing that last example! Much appreciated.
", "rawMarkdown": "Hi! Thanks for the report. I'll chat with the host and see what they'd like to do about these samples.\n\nRe: your post - there are some deeply offensive terms in the text, that are not required to make your point. Please remove the last example / English translation from your post, please.\n\nUpdated to add: thanks for removing that last example! Much appreciated.", "votes": null }, { "id": "865907", "postDate": "05/29/2020 01:20:09", "content": "Thanks your quick feedback.
\nI'm less careful to the content and I updated the post. I'm sorry for that.\nI just picked up samples at random and I had no specific intention.
No worries at all, I thought as much. Just didn't want to leave it there. Thank you!
", "rawMarkdown": "No worries at all, I thought as much. Just didn't want to leave it there. Thank you!", "votes": null }, { "id": "866741", "postDate": "05/29/2020 16:32:11", "content": "Nice find! It's likely that these comments were treated as separate comments during annotation, as they are not exactly the same. In this case they may have been annotated by a different person and therefore they might have different labels in test/validation data.
\n\nIf the BERT ensemble has higher correlation with the average annotator than one annotator with another, then it is understandable that you get a slightly dropped score when using the labels from the validation data.
\n\nGiven the ambiguous nature of toxicity I would not be surprised to see that the BERT ensemble models are more accurate than a human annotator. \nIt is funny that with human annotators and human level AI, this kind of data leak is not such a big concern anymore :)
", "rawMarkdown": "Nice find! It's likely that these comments were treated as separate comments during annotation, as they are not exactly the same. In this case they may have been annotated by a different person and therefore they might have different labels in test/validation data.\n\nIf the BERT ensemble has higher correlation with the average annotator than one annotator with another, then it is understandable that you get a slightly dropped score when using the labels from the validation data. \n\nGiven the ambiguous nature of toxicity I would not be surprised to see that the BERT ensemble models are more accurate than a human annotator. \nIt is funny that with human annotators and human level AI, this kind of data leak is not such a big concern anymore :)", "votes": null }, { "id": "866773", "postDate": "05/29/2020 16:58:18", "content": "Thanks for raising this. We’ve discussed and investigated this with the host, and our conclusion is that we are fine leaving this very minuscule volume of duplicates (relative to the larger test set size) as-is. Some of them are not true dupes, just similar phrasing’s/sets of words or repetitive form replies (suspected to be bot-generated). While others are legitimate dupes, including white space differences.
\n\nSo, the impact of the true dupes remaining, in terms of scoring, is limited. Therefore, we won’t be taking any action at this time and allow them to remain.
", "rawMarkdown": "Thanks for raising this. We’ve discussed and investigated this with the host, and our conclusion is that we are fine leaving this very minuscule volume of duplicates (relative to the larger test set size) as-is. Some of them are not true dupes, just similar phrasing’s/sets of words or repetitive form replies (suspected to be bot-generated). While others are legitimate dupes, including white space differences. \n\nSo, the impact of the true dupes remaining, in terms of scoring, is limited. Therefore, we won’t be taking any action at this time and allow them to remain.", "votes": null }, { "id": "867054", "postDate": "05/30/2020 00:47:15", "content": "Thanks for your good insights and re-posting!
", "rawMarkdown": "Thanks for your good insights and re-posting!", "votes": null }, { "id": "867066", "postDate": "05/30/2020 01:10:17", "content": "Thanks for your clarification. I agree with your conclusions.
\nBut I'm still curious about bot-generated comments, how many, or what kind of, if it truly exists on test data. Because my current approach is not suitable for the detection of bot-generated comments.
\nSo if I find something serious, I will update my post and notebook.
Hi, I was reading the training comments and its full of bot generated comments, like welcome to wikipedia, guidelines how to post images etc, I imagine they will have these in the test too.
", "rawMarkdown": "Hi, I was reading the training comments and its full of bot generated comments, like welcome to wikipedia, guidelines how to post images etc, I imagine they will have these in the test too.", "votes": null }, { "id": "867079", "postDate": "05/30/2020 01:31:00", "content": "Thanks! I will check the training dataset.
\nI assumed there are no bot-generated comments, after reading the paper by wikimedia&jigsaw, https://arxiv.org/abs/1610.08914,
Maybe these are not bot generated or templates but they are repetitive, and a few percent of the comments are similar to these. Or just the filtering is not perfect.
\n\n\n\n\nThis message is regarding the page Bobby Charlton. Thanks for experimenting with Wikipedia. Your test worked, and has been removed. Please use the sandbox for any other tests you want to do. Take a look at the welcome page if you would like to learn more about contributing. Thanks. (spit)
\n
or
\n\n\n", "rawMarkdown": "Maybe these are not bot generated or templates but they are repetitive, and a few percent of the comments are similar to these. Or just the filtering is not perfect.\n\n> This message is regarding the page Bobby Charlton. Thanks for experimenting with Wikipedia. Your test worked, and has been removed. Please use the sandbox for any other tests you want to do. Take a look at the welcome page if you would like to learn more about contributing. Thanks. (spit)\n\nor\n\n> \" Welcome!\n Welcome to Wikipedia, Vivalavida0! My name is {{#if: and I am a Wikipedia administrator| and I am a Wikipedia administrator|}}. I just wanted to say hi and welcome you to Wikipedia! If you have any questions, feel free to leave me a message on my talk page or press the \"\"Request Help\"\" button at the bottom of this message. I love to help new users, so don't be afraid to leave a message! I hope you like the place and decide to stay. \nHere are some pages that will help you edit Wikipedia:\n How to edit a page\n Tutorial\n Sandbox, the place where you can experiment\n Where to ask a question.\n What this site is based on\n Manual of Style \nHere are some ways you can help improve Wikipedia:\n Create an article\n Help remove bad edits (vandalism)\n Translate an article\n Improve illustrations and upload new images\n Perform maintenance tasks\n Become a member of a project that interests you\nI hope you enjoy editing here and being a Wikipedian! Please sign your name on talk pages using four tildes (~~~~); this will automatically produce your name and the date.\nIf you need help, check out Wikipedia:Questions, ask me on my talk page, or press the \"\"Request Help\"\" button below. Again, welcome! {{#if: // have a cup // ark // | // have a cup // ark // |}}", "votes": null }, { "id": "868110", "postDate": "05/31/2020 00:05:07", "content": "\" Welcome!\n Welcome to Wikipedia, Vivalavida0! My name is {{#if: and I am a Wikipedia administrator| and I am a Wikipedia administrator|}}. I just wanted to say hi and welcome you to Wikipedia! If you have any questions, feel free to leave me a message on my talk page or press the \"\"Request Help\"\" button at the bottom of this message. I love to help new users, so don't be afraid to leave a message! I hope you like the place and decide to stay. \n Here are some pages that will help you edit Wikipedia:\n How to edit a page\n Tutorial\n Sandbox, the place where you can experiment\n Where to ask a question.\n What this site is based on\n Manual of Style \n Here are some ways you can help improve Wikipedia:\n Create an article\n Help remove bad edits (vandalism)\n Translate an article\n Improve illustrations and upload new images\n Perform maintenance tasks\n Become a member of a project that interests you\n I hope you enjoy editing here and being a Wikipedian! Please sign your name on talk pages using four tildes (~~~~); this will automatically produce your name and the date.\n If you need help, check out Wikipedia:Questions, ask me on my talk page, or press the \"\"Request Help\"\" button below. Again, welcome! {{#if: // have a cup // ark // | // have a cup // ark // |}}
\n
Thanks for sharing it!\nNow I noticed filtering all bot-generated like comments is really hard. The account information or template filtering should not be enough in this case, there might be partial copy or change.
", "rawMarkdown": "Thanks for sharing it!\nNow I noticed filtering all bot-generated like comments is really hard. The account information or template filtering should not be enough in this case, there might be partial copy or change.", "votes": null }, { "id": "891536", "postDate": "06/18/2020 08:42:51", "content": "I read your notebook https://www.kaggle.com/sai11fkaneko/data-leak, very useful, thanks! \nThought it is strange that there are 4 lang with overlaping, cause validation has only three langs.\nFound out that it is a toxic comment on pt
\n\n\n\n\n'..................................................................................................... ..................................................................................................... >>>>>>>>>NERO IDIOTA IMBECIL.....!!! ZOMBA DE DEUS.. E FAZ MEDIÇÃO DE SEUS PODERES.!!!!!!!!!!!!!!!.UM DIA TU NERO.. TERÁ PROBLEMAS SÉRIOS COMO EU TIVE QUE NEMM DINHEIRO PODIA RESOUVER ..ENTÃO TU VERÁS A GLÓRIA DE DEUS JESUS EM TUA VIDA COMO EU VEJO HOJE EM MINHA VIDA..TU AINDA CONHESSERÁ O ALTAR DA IURD..E PREGARÁ PRAS PESSOAS SOBRE A RIQUEZA INFINITA DE DEUS ..CREIA NISSO!!!!HOJE EU?? OBR.DA IURD.COM MUITO ORGULHO..FAÇO TUDO POR JESUS..!!!E VOCÊ..QUER VER A RIQUEZA DE DEUS..??VIZITA A IURD MAIS PERTO DE TUA CASA..TODAS AS TERÇAS FEIRAS..AS 19:00HRS....E VEJA..A RESPOSTA..!!!!!!!!!!!!!!! ..................................................................................................... .....................................................................................................'
\n
Interesting, how more samples we can find if we remove thats dots
", "rawMarkdown": "I read your notebook https://www.kaggle.com/sai11fkaneko/data-leak, very useful, thanks! \nThought it is strange that there are 4 lang with overlaping, cause validation has only three langs.[](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F788143%2F83423b3ac3776e87616857379e0c7509%2FData-leak-photo.PNG?generation=1592469590241077&alt=media)\nFound out that it is a toxic comment on pt \n > '..................................................................................................... ..................................................................................................... >>>>>>>>>NERO IDIOTA IMBECIL.....!!! ZOMBA DE DEUS.. E FAZ MEDIÇÃO DE SEUS PODERES.!!!!!!!!!!!!!!!.UM DIA TU NERO.. TERÁ PROBLEMAS SÉRIOS COMO EU TIVE QUE NEMM DINHEIRO PODIA RESOUVER ..ENTÃO TU VERÁS A GLÓRIA DE DEUS JESUS EM TUA VIDA COMO EU VEJO HOJE EM MINHA VIDA..TU AINDA CONHESSERÁ O ALTAR DA IURD..E PREGARÁ PRAS PESSOAS SOBRE A RIQUEZA INFINITA DE DEUS ..CREIA NISSO!!!!HOJE EU?? OBR.DA IURD.COM MUITO ORGULHO..FAÇO TUDO POR JESUS..!!!E VOCÊ..QUER VER A RIQUEZA DE DEUS..??VIZITA A IURD MAIS PERTO DE TUA CASA..TODAS AS TERÇAS FEIRAS..AS 19:00HRS....E VEJA..A RESPOSTA..!!!!!!!!!!!!!!! ..................................................................................................... .....................................................................................................'\n\n\n\nInteresting, how more samples we can find if we remove thats dots", "votes": null } ], "comments": [ { "id": 865902, "author_name": "philculliton", "author_url": "", "post_date": "05/29/2020 01:09:03", "content": "Hi! Thanks for the report. I'll chat with the host and see what they'd like to do about these samples.
\n\nRe: your post - there are some deeply offensive terms in the text, that are not required to make your point. Please remove the last example / English translation from your post, please.
\n\nUpdated to add: thanks for removing that last example! Much appreciated.
", "votes": null, "replies": [ { "id": 865907, "author_name": "sai11fkaneko", "author_url": "", "post_date": "05/29/2020 01:20:09", "content": "Thanks your quick feedback.
\nI'm less careful to the content and I updated the post. I'm sorry for that.\nI just picked up samples at random and I had no specific intention.
No worries at all, I thought as much. Just didn't want to leave it there. Thank you!
", "votes": null, "replies": [] } ] }, { "id": 866741, "author_name": "riblidezso", "author_url": "", "post_date": "05/29/2020 16:32:11", "content": "Nice find! It's likely that these comments were treated as separate comments during annotation, as they are not exactly the same. In this case they may have been annotated by a different person and therefore they might have different labels in test/validation data.
\n\nIf the BERT ensemble has higher correlation with the average annotator than one annotator with another, then it is understandable that you get a slightly dropped score when using the labels from the validation data.
\n\nGiven the ambiguous nature of toxicity I would not be surprised to see that the BERT ensemble models are more accurate than a human annotator. \nIt is funny that with human annotators and human level AI, this kind of data leak is not such a big concern anymore :)
", "votes": null, "replies": [ { "id": 867054, "author_name": "sai11fkaneko", "author_url": "", "post_date": "05/30/2020 00:47:15", "content": "Thanks for your good insights and re-posting!
", "votes": null, "replies": [] } ] }, { "id": 866773, "author_name": "juliaelliott", "author_url": "", "post_date": "05/29/2020 16:58:18", "content": "Thanks for raising this. We’ve discussed and investigated this with the host, and our conclusion is that we are fine leaving this very minuscule volume of duplicates (relative to the larger test set size) as-is. Some of them are not true dupes, just similar phrasing’s/sets of words or repetitive form replies (suspected to be bot-generated). While others are legitimate dupes, including white space differences.
\n\nSo, the impact of the true dupes remaining, in terms of scoring, is limited. Therefore, we won’t be taking any action at this time and allow them to remain.
", "votes": null, "replies": [ { "id": 867066, "author_name": "sai11fkaneko", "author_url": "", "post_date": "05/30/2020 01:10:17", "content": "Thanks for your clarification. I agree with your conclusions.
\nBut I'm still curious about bot-generated comments, how many, or what kind of, if it truly exists on test data. Because my current approach is not suitable for the detection of bot-generated comments.
\nSo if I find something serious, I will update my post and notebook.
Hi, I was reading the training comments and its full of bot generated comments, like welcome to wikipedia, guidelines how to post images etc, I imagine they will have these in the test too.
", "votes": null, "replies": [] }, { "id": 867079, "author_name": "sai11fkaneko", "author_url": "", "post_date": "05/30/2020 01:31:00", "content": "Thanks! I will check the training dataset.
\nI assumed there are no bot-generated comments, after reading the paper by wikimedia&jigsaw, https://arxiv.org/abs/1610.08914,
Maybe these are not bot generated or templates but they are repetitive, and a few percent of the comments are similar to these. Or just the filtering is not perfect.
\n\n\n\n\nThis message is regarding the page Bobby Charlton. Thanks for experimenting with Wikipedia. Your test worked, and has been removed. Please use the sandbox for any other tests you want to do. Take a look at the welcome page if you would like to learn more about contributing. Thanks. (spit)
\n
or
\n\n\n", "votes": null, "replies": [] }, { "id": 868110, "author_name": "sai11fkaneko", "author_url": "", "post_date": "05/31/2020 00:05:07", "content": "\" Welcome!\n Welcome to Wikipedia, Vivalavida0! My name is {{#if: and I am a Wikipedia administrator| and I am a Wikipedia administrator|}}. I just wanted to say hi and welcome you to Wikipedia! If you have any questions, feel free to leave me a message on my talk page or press the \"\"Request Help\"\" button at the bottom of this message. I love to help new users, so don't be afraid to leave a message! I hope you like the place and decide to stay. \n Here are some pages that will help you edit Wikipedia:\n How to edit a page\n Tutorial\n Sandbox, the place where you can experiment\n Where to ask a question.\n What this site is based on\n Manual of Style \n Here are some ways you can help improve Wikipedia:\n Create an article\n Help remove bad edits (vandalism)\n Translate an article\n Improve illustrations and upload new images\n Perform maintenance tasks\n Become a member of a project that interests you\n I hope you enjoy editing here and being a Wikipedian! Please sign your name on talk pages using four tildes (~~~~); this will automatically produce your name and the date.\n If you need help, check out Wikipedia:Questions, ask me on my talk page, or press the \"\"Request Help\"\" button below. Again, welcome! {{#if: // have a cup // ark // | // have a cup // ark // |}}
\n
Thanks for sharing it!\nNow I noticed filtering all bot-generated like comments is really hard. The account information or template filtering should not be enough in this case, there might be partial copy or change.
", "votes": null, "replies": [] } ] }, { "id": 891536, "author_name": "aybatov", "author_url": "", "post_date": "06/18/2020 08:42:51", "content": "I read your notebook https://www.kaggle.com/sai11fkaneko/data-leak, very useful, thanks! \nThought it is strange that there are 4 lang with overlaping, cause validation has only three langs.\nFound out that it is a toxic comment on pt
\n\n\n\n\n'..................................................................................................... ..................................................................................................... >>>>>>>>>NERO IDIOTA IMBECIL.....!!! ZOMBA DE DEUS.. E FAZ MEDIÇÃO DE SEUS PODERES.!!!!!!!!!!!!!!!.UM DIA TU NERO.. TERÁ PROBLEMAS SÉRIOS COMO EU TIVE QUE NEMM DINHEIRO PODIA RESOUVER ..ENTÃO TU VERÁS A GLÓRIA DE DEUS JESUS EM TUA VIDA COMO EU VEJO HOJE EM MINHA VIDA..TU AINDA CONHESSERÁ O ALTAR DA IURD..E PREGARÁ PRAS PESSOAS SOBRE A RIQUEZA INFINITA DE DEUS ..CREIA NISSO!!!!HOJE EU?? OBR.DA IURD.COM MUITO ORGULHO..FAÇO TUDO POR JESUS..!!!E VOCÊ..QUER VER A RIQUEZA DE DEUS..??VIZITA A IURD MAIS PERTO DE TUA CASA..TODAS AS TERÇAS FEIRAS..AS 19:00HRS....E VEJA..A RESPOSTA..!!!!!!!!!!!!!!! ..................................................................................................... .....................................................................................................'
\n
Interesting, how more samples we can find if we remove thats dots
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "865850": "There might be leaked samples from validation data, ~600 samples on test data.\nI could confirm some minor changed validation samples on test data or the exact same samples on test data like below.\nI displayed these samples on the notebook (https://www.kaggle.com/sai11fkaneko/data-leak )\n\nAnd also I tried whether LB score is improved. But I got worse result if I use this leak.(0.9462 -> 0.9459: base submission file from https://www.kaggle.com/hamditarek/ensemble )\nOne possible reason for this is my rough treatment after 128 tokens. \nBut still not sure for the reason...\n```\n--------------------------------------------------------------------------------\n>> TEST \n ID:8667, LANG:tr, Duplicated Num:2\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi . Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.\n\n>> VALIDATION \n ID:2162, LANG:tr, Duplicated Num:1, TOXIC:0\nCOMMET_TEXT:Vikipedi de deneme yaptığınız için teşekkürler. Denemeniz çalıştı, ancak şu anda ya geri alındı ya da silindi. Başka bir deneme yapmak istiyorsanız lütfen deneme tahtasını kullanın. Ansiklopedimize nasıl katkıda bulunabileceğiniz hakkında daha fazla bilgi edinmek istiyorsanız hoşgeldin sayfasına bir göz atın. Teşekkürler.\n\n>> diff result\n*** TEST\n--- VAL\n***************\n*** 14,21 ****\n alındı ya da! silindi! . Başka bir deneme--- 14,20 ----\n alındı ya da! silindi. Başka bir deneme\n\n>> TEST ENGLISH Translation\nThank you for doing experiments in Wikipedia. Your experiment worked, but at the moment either rolled back or deleted. If you want to make another attempt, please use the sandbox. If you want to learn more about how you might be contributing to our encyclopedia Take a look at the welcome page. Thank you.\n\n```", "865902": "Hi! Thanks for the report. I'll chat with the host and see what they'd like to do about these samples.\n\nRe: your post - there are some deeply offensive terms in the text, that are not required to make your point. Please remove the last example / English translation from your post, please.\n\nUpdated to add: thanks for removing that last example! Much appreciated.", "865907": "Thanks your quick feedback. \nI'm less careful to the content and I updated the post. I'm sorry for that.\nI just picked up samples at random and I had no specific intention.", "865909": "No worries at all, I thought as much. Just didn't want to leave it there. Thank you!", "866741": "Nice find! It's likely that these comments were treated as separate comments during annotation, as they are not exactly the same. In this case they may have been annotated by a different person and therefore they might have different labels in test/validation data.\n\nIf the BERT ensemble has higher correlation with the average annotator than one annotator with another, then it is understandable that you get a slightly dropped score when using the labels from the validation data. \n\nGiven the ambiguous nature of toxicity I would not be surprised to see that the BERT ensemble models are more accurate than a human annotator. \nIt is funny that with human annotators and human level AI, this kind of data leak is not such a big concern anymore :)", "866773": "Thanks for raising this. We’ve discussed and investigated this with the host, and our conclusion is that we are fine leaving this very minuscule volume of duplicates (relative to the larger test set size) as-is. Some of them are not true dupes, just similar phrasing’s/sets of words or repetitive form replies (suspected to be bot-generated). While others are legitimate dupes, including white space differences. \n\nSo, the impact of the true dupes remaining, in terms of scoring, is limited. Therefore, we won’t be taking any action at this time and allow them to remain.", "867054": "Thanks for your good insights and re-posting!", "867066": "Thanks for your clarification. I agree with your conclusions. \nBut I'm still curious about bot-generated comments, how many, or what kind of, if it truly exists on test data. Because my current approach is not suitable for the detection of bot-generated comments. \nSo if I find something serious, I will update my post and notebook.", "867070": "Hi, I was reading the training comments and its full of bot generated comments, like welcome to wikipedia, guidelines how to post images etc, I imagine they will have these in the test too.", "867079": "Thanks! I will check the training dataset. \nI assumed there are no bot-generated comments, after reading the paper by wikimedia&jigsaw, https://arxiv.org/abs/1610.08914, \n\n", "867089": "Maybe these are not bot generated or templates but they are repetitive, and a few percent of the comments are similar to these. Or just the filtering is not perfect.\n\n> This message is regarding the page Bobby Charlton. Thanks for experimenting with Wikipedia. Your test worked, and has been removed. Please use the sandbox for any other tests you want to do. Take a look at the welcome page if you would like to learn more about contributing. Thanks. (spit)\n\nor\n\n> \" Welcome!\n Welcome to Wikipedia, Vivalavida0! My name is {{#if: and I am a Wikipedia administrator| and I am a Wikipedia administrator|}}. I just wanted to say hi and welcome you to Wikipedia! If you have any questions, feel free to leave me a message on my talk page or press the \"\"Request Help\"\" button at the bottom of this message. I love to help new users, so don't be afraid to leave a message! I hope you like the place and decide to stay. \nHere are some pages that will help you edit Wikipedia:\n How to edit a page\n Tutorial\n Sandbox, the place where you can experiment\n Where to ask a question.\n What this site is based on\n Manual of Style \nHere are some ways you can help improve Wikipedia:\n Create an article\n Help remove bad edits (vandalism)\n Translate an article\n Improve illustrations and upload new images\n Perform maintenance tasks\n Become a member of a project that interests you\nI hope you enjoy editing here and being a Wikipedian! Please sign your name on talk pages using four tildes (~~~~); this will automatically produce your name and the date.\nIf you need help, check out Wikipedia:Questions, ask me on my talk page, or press the \"\"Request Help\"\" button below. Again, welcome! {{#if: // have a cup // ark // | // have a cup // ark // |}}", "868110": "Thanks for sharing it!\nNow I noticed filtering all bot-generated like comments is really hard. The account information or template filtering should not be enough in this case, there might be partial copy or change.", "891536": "I read your notebook https://www.kaggle.com/sai11fkaneko/data-leak, very useful, thanks! \nThought it is strange that there are 4 lang with overlaping, cause validation has only three langs.[](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F788143%2F83423b3ac3776e87616857379e0c7509%2FData-leak-photo.PNG?generation=1592469590241077&alt=media)\nFound out that it is a toxic comment on pt \n > '..................................................................................................... ..................................................................................................... >>>>>>>>>NERO IDIOTA IMBECIL.....!!! ZOMBA DE DEUS.. E FAZ MEDIÇÃO DE SEUS PODERES.!!!!!!!!!!!!!!!.UM DIA TU NERO.. TERÁ PROBLEMAS SÉRIOS COMO EU TIVE QUE NEMM DINHEIRO PODIA RESOUVER ..ENTÃO TU VERÁS A GLÓRIA DE DEUS JESUS EM TUA VIDA COMO EU VEJO HOJE EM MINHA VIDA..TU AINDA CONHESSERÁ O ALTAR DA IURD..E PREGARÁ PRAS PESSOAS SOBRE A RIQUEZA INFINITA DE DEUS ..CREIA NISSO!!!!HOJE EU?? OBR.DA IURD.COM MUITO ORGULHO..FAÇO TUDO POR JESUS..!!!E VOCÊ..QUER VER A RIQUEZA DE DEUS..??VIZITA A IURD MAIS PERTO DE TUA CASA..TODAS AS TERÇAS FEIRAS..AS 19:00HRS....E VEJA..A RESPOSTA..!!!!!!!!!!!!!!! ..................................................................................................... .....................................................................................................'\n\n\n\nInteresting, how more samples we can find if we remove thats dots" }, "source": "meta" }![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F788143%2F83423b3ac3776e87616857379e0c7509%2FData-leak-photo.PNG?generation=1592469590241077&alt=media\"){kind=link}