{"cells":[{"metadata":{},"cell_type":"markdown","source":"

Jigsaw Multilingual Toxic Comment Classification

\n

EDA

\n\n\n[Jigsaw Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/overview) goal is to take advantage of [Kaggle's new TPU support](https://www.kaggle.com/docs/tpu) to build multilingual models with English-only training data. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Acknowledgements ❤ \n\nThis notebook was generated with compiling the best bits of all the best EDA notebooks. Thanks to them!\n\n1. [Jigsaw Multilingual: Quick EDA & TPU Modeling](https://www.kaggle.com/ipythonx/jigsaw-multilingual-quick-eda-tpu-modeling)\n2. [Jigsaw Multilingual Toxicity : EDA + Models](https://www.kaggle.com/tarunpaparaju/jigsaw-multilingual-toxicity-eda-models)\n3. [Stop the S@# - Toxic Comments EDA](https://www.kaggle.com/jagangupta/stop-the-s-toxic-comments-eda/notebook)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n- Introduction \n- Load Libraries \n- Prepare Data for EDA \n- EDA \n - Example Comments \n - Distribution of Toxicity \n - Languages \n - Wordclouds - Frequent Words \n - EDA of Indirect Features \n - Distribution of Characters & Words \n - Sentiment vs. Toxicity: Sentiment Analysis of Comment Toxicity \n - Negative Sentiment \n - Positive Sentiment \n - Neutral Sentiment \n - Compound Sentiment ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Introduction \n\n**What should we expect the data format to be?**\n\n> The primary data for the competition is, in each provided file, the `comment_text` column. This contains the text of a comment which has been classified as `toxic` or non-toxic (0...1 in the toxic column). The train set’s comments are entirely in english and come either from Civil Comments or Wikipedia talk page edits. The test data's `comment_text` columns are composed of multiple non-English languages.\nThe `*-train.csv` files and `validation.csv` file also contain a toxic column that is the target to be trained on. \n\n> The `jigsaw-toxic-comment`-train.csv and `jigsaw-unintended-bias-train.csv` contain training data (`comment_text` and `toxic`) from the two previous Jigsaw competitions, as well as additional columns that you may find useful. `*-seqlen128.csv` files contain training, validation, and test data that has been processed for input into BERT.\n\n**What am I predicting?**\n\n> You are predicting the probability that a comment is `toxic`. A toxic comment would receive a `1.0`. A benign, `non-toxic` comment would receive a `0.0`. In the `test set`, all comments are classified as either a `1.0` or a `0.0`.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Load Libraries ","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport plotly.figure_factory as ff\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport os\nimport re\nfrom tqdm import tqdm\ntqdm.pandas()\n\n\nfrom wordcloud import WordCloud, STOPWORDS\nfrom PIL import Image\nfrom kaggle_datasets import KaggleDatasets\nfrom colorama import Fore, Back, Style, init\nimport plotly.graph_objects as go\n\nimport nltk\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare Data for EDA \n\n**Most Important Files**\n- `jigsaw-toxic-comment-train.csv`: data from [this competition ](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge)\n- `jigsaw-unintended-bias-train.csv`: data from [this competition](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification)\n- `validation.csv`: comments from Wikipedia talk pages in different non-English languages\n- `test.csv`: comments from Wikipedia talk pages in different non-English languages\n- `sample_submission.csv`: a sample submission file in the correct format","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"dir = '/kaggle/input/jigsaw-multilingual-toxic-comment-classification'\n\ntrain_set1 = pd.read_csv(os.path.join(dir, 'jigsaw-toxic-comment-train.csv'))\ntrain_set2 = pd.read_csv(os.path.join(dir, 'jigsaw-unintended-bias-train.csv'))\ntrain_set2.toxic = train_set2.toxic.round().astype(int)\n\nvalid = pd.read_csv(os.path.join(dir, 'validation.csv'))\ntest = pd.read_csv(os.path.join(dir, 'test.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Combine train1 with a subset of train2\ntrain = pd.concat([\n train_set1[['comment_text', 'toxic']],\n train_set2[['comment_text', 'toxic']].query('toxic==1'),\n train_set2[['comment_text', 'toxic']].query('toxic==0').sample(n=100000, random_state=0)\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(valid.shape)\nvalid.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Check for missing values in Train dataset\")\nnull_check=train.isnull().sum()\nprint(null_check)\nprint(\"Check for missing values in Validation dataset\")\nnull_check=valid.isnull().sum()\nprint(null_check)\nprint(\"Check for missing values in Test dataset\")\nnull_check=test.isnull().sum()\nprint(null_check)\nprint(\"filling NA with \\\"unknown\\\"\")\ntrain[\"comment_text\"].fillna(\"unknown\", inplace=True)\nvalid[\"comment_text\"].fillna(\"unknown\", inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"****\n## Example Comments ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(3):\n print(f'[Comment {i+1}]\\n', train['comment_text'][i])\n print()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Toxic comments:\")\nprint(train[train.toxic==1].iloc[:10,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"****\n## Distribution of Toxicity \n\nAnd so we observe following columns so far.\n\n- `id` - identifier within each file.\n- `comment_text` - the text of the comment to be classified.\n- `lang` - the language of the comment.\n- `toxic` - whether or not the comment is classified as toxic. (Does not exist in `test.csv`.)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#print(train.toxic.value_counts())\n#print(valid.toxic.value_counts())\n\nprint(\"Train set\")\nprint(\"Toxic comments = \",len(train[train['toxic']==1]))\nprint(\"Non-toxic comments = \",len(train[train['toxic']==0]))\n\nprint(\"\\nValidation set\")\nprint(\"Toxic comments = \",len(valid[valid['toxic']==1]))\nprint(\"Non-toxic comments = \",len(valid[valid['toxic']==0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set(style=\"darkgrid\")\n\nf = plt.figure(figsize=(20,5))\nf.add_subplot(1,2,1)\nsns.countplot(train_set1.toxic)\nplt.title('Toxic Comment Distribution in Train Set 1')\nf.add_subplot(1,2,2)\nsns.countplot(train_set2.toxic)\nplt.title('Toxic Comment Distribution in Train Set 2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- We can see that there is a huge imbalance in the data. Therefore, it made sense to make new train set, combining Train Sets 1 & 2, but downsampling the non-toxic comments from Train Set 2. Below you can see the slightly more balanced Train Set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(20,5))\nf.add_subplot(1,2,1)\nsns.countplot(train.toxic)\nplt.title('Toxic Comment Distribution in Train Set')\nf.add_subplot(1,2,2)\nsns.countplot(valid.toxic)\nplt.title('Toxic Comment Distribution in Validation Set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a clear class imbalance, which could lead to bias towards a comment being classified as 0 (non-toxic). We can manage this bias by preprocessing the data with upsampling-downsampling.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"****\n## Languages ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(valid.lang.value_counts())\nprint(test.lang.value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(20,5))\nf.add_subplot(1,2,1)\nsns.countplot(valid.lang)\nplt.title('Langauages in Validation Set')\nf.add_subplot(1,2,2)\nsns.countplot(test.lang)\nplt.title('Languages in Final Test Set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"****\n## Wordclouds - Frequent Words ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"stopword=set(STOPWORDS)\n\n#wordcloud of all comments\nplt.figure(figsize=(10,10))\ntext = train.comment_text.values\nwc = WordCloud(background_color=\"black\",max_words=2000,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Common words in All Comments\", fontsize=16)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#non-toxic wordcloud\nclean_mask=np.array(Image.open(\"../input/imagesforkernal/safe-zone.png\"))\nclean_mask=clean_mask[:,:,1]\n\nplt.figure(figsize=(20,20))\nplt.subplot(121)\nsubset = train.query(\"toxic == 0\")\ntext = subset.comment_text.values\nwc = WordCloud(background_color=\"black\",max_words=1000,mask=clean_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Common words in non-Toxic Comments\", fontsize=16)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)\n\n#toxic wordcloud\nclean_mask=np.array(Image.open(\"../input/imagesforkernal/swords.png\"))\nclean_mask=clean_mask[:,:,1]\n\nplt.subplot(122)\nsubset = train.query(\"toxic == 1\")\ntext = subset.comment_text.values\nwc = WordCloud(background_color=\"black\",max_words=1000,mask=clean_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Common words in Toxic Comments\", fontsize=16)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"****\n## EDA of Indirect Features \n\n\n- count of sentences\n- count of words\n- count of unique words\n- count of characters\n- count of punctuations\n- count of uppercase words/letters\n- count of stop words\n- avg length of each word","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Characters & Words ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nums_1 = train[train['toxic']==1]['comment_text'].sample(frac=0.1).str.len()\nnums_2 = train[train['toxic']==0]['comment_text'].sample(frac=0.1).str.len()\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"red\", \"green\"], show_hist=False)\n\nfig.update_layout(title_text=\"Number of characters per comment vs. Toxicity\", xaxis_title=\"No of characters per comment\", \n yaxis_title=\"Distribution of observations (%)\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nums_1 = train[train['toxic']==1]['comment_text'].sample(frac=0.1).str.split().str.len()\nnums_2 = train[train['toxic']==0]['comment_text'].sample(frac=0.1).str.split().str.len()\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"red\", \"green\"], show_hist=False)\n\nfig.update_layout(title_text=\"Number of words per comment vs. Toxicity\", xaxis_title=\"No of words per comment\", \n yaxis_title=\"Distribution of observations (%)\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Toxic comments typically have more characters & words.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"****\n## Sentiment vs. Toxicity: Sentiment Analysis of Comment Toxicity \n\n**Do _Sentiment_ and _Toxicity_ have a relationship? We could hypothesize that if a comment is a higher negative sentiment, then it is more likely to be toxic.**\n\nSentiment and polarity are quantities that reflect the emotion and intention behind a sentence. Now, I will give a sentiment intensity score to comments using the NLTK (natural language toolkit) library.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"SIA = SentimentIntensityAnalyzer()\n\ndef polarity(x):\n if type(x) == str:\n return SIA.polarity_scores(x)\n else:\n return 1000\n \ntrain[\"polarity\"] = train[\"comment_text\"].progress_apply(polarity)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train[train.toxic==1].iloc[4,0])\n\npolarity(train[train.toxic==1].iloc[4,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Positive, Negative and Neutral scores represent the proportion of text that falls in these categories. This means our sentence was rated as 0% Positive, 45% Neutral and 55% Negative. Hence all these should add up to 1.\n\nFrom this comment sentiment example alone we can already see that sentiment scores are not a reliable reflection of how toxic a comment is. This comment is clearly toxic, however it has only been rated as 55% negative. Would you say this comment is only 55% toxic? Probably not... Let's explore the sentiment analyis any way.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Negative Sentiment \nNegative sentiment refers to negative or pessimistic emotions. It is a score between 0 and 1; the greater the score, the more negative the abstract is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neg\"] for pols in train[\"polarity\"] if pols[\"neg\"] != 0], marker=dict(color='red')))\nfig.update_layout(xaxis_title=\"Negative sentiment\", title_text=\"Negative sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Most comments have a low negativity sentiment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"negativity\"] = train[\"polarity\"].apply(lambda x: x[\"neg\"])\n\nnums_1 = train.sample(frac=0.1).query(\"toxic == 1\")[\"negativity\"]\nnums_2 = train.sample(frac=0.1).query(\"toxic == 0\")[\"negativity\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-Toxic\"],\n colors=[\"red\", \"green\"], show_hist=False)\n\nfig.update_layout(title_text=\"Negative Sentiment vs. Toxicity\", xaxis_title=\"Negative Sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Comments with low negative sentiment are more likely to be non-toxic, and comments with high negative sentiment are more likely to be toxic. \n- A comment is likely to be non-toxic if it has a negativity of 0.\n- A comment is likely to be toxic if it has a negativity more than 0.8.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Positive Sentiment \nPositive sentiment refers to positive or optimistic emotions. It is a score between 0 and 1; the greater the score, the more positive the abstract is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"pos\"] for pols in train[\"polarity\"] if pols[\"pos\"] != 0], marker=dict(color='green')))\nfig.update_layout(xaxis_title=\"Positive sentiment\", title_text=\"Positive sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Most comments have low positive sentiment.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"positivity\"] = train[\"polarity\"].apply(lambda x: x[\"pos\"])\n\nnums_1 = train.sample(frac=0.1).query(\"toxic == 1\")[\"positivity\"]\nnums_2 = train.sample(frac=0.1).query(\"toxic == 0\")[\"positivity\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-Toxic\"],\n colors=[\"red\", \"green\"], show_hist=False)\n\nfig.update_layout(title_text=\"Positive Sentiment vs. Toxicity\", xaxis_title=\"Positive Sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- The higher the positivity sentiment, the more likely the comment is non-toxic. \n- However, we can see that both the distributions are very similar, indicating that positive sentiment is not an accurate indicator of comment toxicity.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Neutral Sentiment \nNeutrality sentiment refers to the level of bias or opinion in the text. It is a score between 0 and 1; the greater the score, the more neutral/unbiased the abstract is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neu\"] for pols in train[\"polarity\"] if pols[\"neu\"] != 1], marker=dict(color='grey')))\nfig.update_layout(xaxis_title=\"Neutral sentiment\", title_text=\"Neutral sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Most comments are neutral -- meaning that they are unopinionated or unbiased. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"neutral\"] = train[\"polarity\"].apply(lambda x: x[\"neu\"])\n\nnums_1 = train.sample(frac=0.1).query(\"toxic == 1\")[\"neutral\"]\nnums_2 = train.sample(frac=0.1).query(\"toxic == 0\")[\"neutral\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-Toxic\"],\n colors=[\"red\", \"green\"], show_hist=False)\n\nfig.update_layout(title_text=\"Neutral Sentiment vs. Toxicity\", xaxis_title=\"Neutral Sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- A comment with a neutral sentiment of exactly 1 is likely a non-toxic comment. This is beause the probability density of the non-toxic distribution experiences a sudden jump at 1, and the probability density of the toxic distribution is significantly lower at the same point. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Compound Sentiment \nCompoundness sentiment refers to the total level of sentiment in the sentence. It is a score between -1 and 1; the greater the score, the more emotional the abstract is.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"compound\"] for pols in train[\"polarity\"] if pols[\"compound\"] != 0], marker=dict(color='yellow')))\nfig.update_layout(xaxis_title=\"Compound sentiment\", title_text=\"Compound sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- All comments are evenly distribution among different levels of compound sentiment, meaning the comments express a variety of emotions.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"compound\"] = train[\"polarity\"].apply(lambda x: x[\"compound\"])\n\nnums_1 = train.sample(frac=0.1).query(\"toxic == 1\")[\"compound\"]\nnums_2 = train.sample(frac=0.1).query(\"toxic == 0\")[\"compound\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-Toxic\"],\n colors=[\"red\", \"green\"], show_hist=False)\n\nfig.update_layout(title_text=\"Compound Sentiment vs. Toxicity\", xaxis_title=\"Compound Sentiment\", \n yaxis_title=\"Number of comments\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Non-toxic comments have a higher compound sentiment.\n- Toxic comments have a lower compound sentiment.\n- Compound sentiment (compared to Negative, Positive, Neutral sentiments) seems do have a greater visible correlation with toxicity.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The next step is to build models!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}