I found decent amont of insincere questions contain pure toxicity. Recently, Kaggle hosted such competition, Toxic Comment Classification where the toxic list was among winning recipe along with fastText trained on common crawl (I miss it. Kaggle team can you consider uploading fastText trained on common crawl instead of Word2Vec?).
\n\nHere are few resources related to making meta-features, list of toxic words...etc. I will be updating them over the period of competition.
\n\nI found decent amont of insincere questions contain pure toxicity. Recently, Kaggle hosted such competition, Toxic Comment Classification where the toxic list was among winning recipe along with fastText trained on common crawl (I miss it. Kaggle team can you consider uploading fastText trained on common crawl instead of Word2Vec?).
\n\nHere are few resources related to making meta-features, list of toxic words...etc. I will be updating them over the period of competition.
\n\nHi, Shahebaz. Once again a very good job!\nThe feature engineering links and previous solution might be especially helpful for those which are not so much into Deep Learning. But I would say, the \"bad / swear word list\" is not allowed according to the competition rules. Because it is external data. But If one wants to risk using it, I would recommend to take the top x words with the best normalized insincere / sincere occurrence ratio. That gives a pretty good correlation as I have found out - before I remembered that external data isn't allowed.
", "rawMarkdown": "Hi, Shahebaz. Once again a very good job!\nThe feature engineering links and previous solution might be especially helpful for those which are not so much into Deep Learning. But I would say, the \"bad / swear word list\" is not allowed according to the competition rules. Because it is external data. But If one wants to risk using it, I would recommend to take the top x words with the best normalized insincere / sincere occurrence ratio. That gives a pretty good correlation as I have found out - before I remembered that external data isn't allowed.", "votes": null }, { "id": "439956", "postDate": "12/16/2018 18:03:09", "content": "I second you. We are using contractions_mapping and extracted punctuations. And these are allowed for the competition as these are kernel generated lists. But, using direct external list as such this is out of rules and not allowed (IMO).
\n\n> Yes. You can utilize any data that is generated during the Kernel run.
\n\nTo get a way around this. You can extract similar words that begin with \"bad_word\" extract its embedding and take top n similar words and make a list. Thus, that becomes kernel generated using given data. Using the list linked in this thread, teams can test whether toxic words features works? And if works gives how much F1 score boost? And figure out features that can be generated within dataset and in accordance to rules. Thanks :)
", "rawMarkdown": "I second you. We are using contractions_mapping and extracted punctuations. And these **are allowed for the competition** as these are kernel generated lists. But, using direct external list as such this is **out of rules and not allowed** (IMO).\n\n> [Yes. You can utilize any data that is generated during the Kernel run.][1]\n\n\n\nTo get a way around this. You can extract similar words that begin with \"bad_word\" extract its embedding and take top n similar words and make a list. Thus, that becomes kernel generated using given data. Using the list linked in this thread, teams can test whether toxic words features works? And if works gives how much F1 score boost? And figure out features that can be generated within dataset and in accordance to rules. Thanks :) \n\n\n [1]: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/70715#417913", "votes": null }, { "id": "439974", "postDate": "12/16/2018 18:54:34", "content": "I agree. Although I read the \"utilize any data [..] generated during Kernel run\" as: \"You are not allowed to save this as result from another Kernel but have to calculate it live.\" So only features which you can calculate within a feasible time frame are a good idea in the first place. I actually asked a question in this direction but it got never answered.\nSo the \"bad word\" approach still wouldn't be allowed. Because you have to have a ground truth somewhere. The only way out of it would therefore probably be a sentiment analysis for words only. SpaCy should have one. But this however will take a long time to compute.
", "rawMarkdown": "I agree. Although I read the \"utilize any data [..] generated during Kernel run\" as: \"You are not allowed to save this as result from another Kernel but have to calculate it live.\" So only features which you can calculate within a feasible time frame are a good idea in the first place. I actually asked a question in this direction but it got never answered.\nSo the \"bad word\" approach still wouldn't be allowed. Because you have to have a ground truth somewhere. The only way out of it would therefore probably be a sentiment analysis for words only. SpaCy should have one. But this however will take a long time to compute.", "votes": null }, { "id": "3146987", "postDate": "03/11/2025 14:30:30", "content": "any update for this dataset
", "rawMarkdown": "any update for this dataset", "votes": null } ], "comments": [ { "id": 3146987, "author_name": "borhaneddineboudiaf", "author_url": "", "post_date": "03/11/2025 14:30:30", "content": "any update for this dataset
", "votes": null, "replies": [] }, { "id": 439945, "author_name": "cepheidq", "author_url": "", "post_date": "12/16/2018 17:51:06", "content": "Hi, Shahebaz. Once again a very good job!\nThe feature engineering links and previous solution might be especially helpful for those which are not so much into Deep Learning. But I would say, the \"bad / swear word list\" is not allowed according to the competition rules. Because it is external data. But If one wants to risk using it, I would recommend to take the top x words with the best normalized insincere / sincere occurrence ratio. That gives a pretty good correlation as I have found out - before I remembered that external data isn't allowed.
", "votes": null, "replies": [ { "id": 439956, "author_name": "shaz13", "author_url": "", "post_date": "12/16/2018 18:03:09", "content": "I second you. We are using contractions_mapping and extracted punctuations. And these are allowed for the competition as these are kernel generated lists. But, using direct external list as such this is out of rules and not allowed (IMO).
\n\n> Yes. You can utilize any data that is generated during the Kernel run.
\n\nTo get a way around this. You can extract similar words that begin with \"bad_word\" extract its embedding and take top n similar words and make a list. Thus, that becomes kernel generated using given data. Using the list linked in this thread, teams can test whether toxic words features works? And if works gives how much F1 score boost? And figure out features that can be generated within dataset and in accordance to rules. Thanks :)
", "votes": null, "replies": [] }, { "id": 439974, "author_name": "cepheidq", "author_url": "", "post_date": "12/16/2018 18:54:34", "content": "I agree. Although I read the \"utilize any data [..] generated during Kernel run\" as: \"You are not allowed to save this as result from another Kernel but have to calculate it live.\" So only features which you can calculate within a feasible time frame are a good idea in the first place. I actually asked a question in this direction but it got never answered.\nSo the \"bad word\" approach still wouldn't be allowed. Because you have to have a ground truth somewhere. The only way out of it would therefore probably be a sentiment analysis for words only. SpaCy should have one. But this however will take a long time to compute.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "439823": "I found decent amont of insincere questions contain pure toxicity. Recently, Kaggle hosted such competition, [Toxic Comment Classification][1] where the toxic list was among winning recipe along with fastText trained on common crawl (_I miss it. Kaggle team can you consider uploading fastText trained on common crawl instead of Word2Vec?_).\n\nHere are few resources related to making meta-features, list of toxic words...etc. I will be updating them over the period of competition. \n\n\n# Resources\n\n1. [Full List of Bad and Swear Words Banned by Google][4]\n2. [Open Solution of Toxic Comment Challenge][3] (lots of models, functions in there )\n3. [Understanding Feature Engineering — Traditional Methods for Text Data][5]\n4. [How Toxic Are Quora Questions?][2] @tunguz \n\n\n [1]: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/\n [2]: https://www.kaggle.com/tunguz/how-toxic-are-quora-questions\n [3]: https://github.com/neptune-ml/open-solution-toxic-comments\n [4]: https://github.com/RobertJGabriel/Google-profanity-words/blob/master/list.txt\n [5]: https://towardsdatascience.com/understanding-feature-engineering-part-3-traditional-methods-for-text-data-f6f7d70acd41", "439945": "Hi, Shahebaz. Once again a very good job!\nThe feature engineering links and previous solution might be especially helpful for those which are not so much into Deep Learning. But I would say, the \"bad / swear word list\" is not allowed according to the competition rules. Because it is external data. But If one wants to risk using it, I would recommend to take the top x words with the best normalized insincere / sincere occurrence ratio. That gives a pretty good correlation as I have found out - before I remembered that external data isn't allowed.", "439956": "I second you. We are using contractions_mapping and extracted punctuations. And these **are allowed for the competition** as these are kernel generated lists. But, using direct external list as such this is **out of rules and not allowed** (IMO).\n\n> [Yes. You can utilize any data that is generated during the Kernel run.][1]\n\n\n\nTo get a way around this. You can extract similar words that begin with \"bad_word\" extract its embedding and take top n similar words and make a list. Thus, that becomes kernel generated using given data. Using the list linked in this thread, teams can test whether toxic words features works? And if works gives how much F1 score boost? And figure out features that can be generated within dataset and in accordance to rules. Thanks :) \n\n\n [1]: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/70715#417913", "439974": "I agree. Although I read the \"utilize any data [..] generated during Kernel run\" as: \"You are not allowed to save this as result from another Kernel but have to calculate it live.\" So only features which you can calculate within a feasible time frame are a good idea in the first place. I actually asked a question in this direction but it got never answered.\nSo the \"bad word\" approach still wouldn't be allowed. Because you have to have a ground truth somewhere. The only way out of it would therefore probably be a sentiment analysis for words only. SpaCy should have one. But this however will take a long time to compute.", "3146987": "any update for this dataset" }, "source": "meta" }