{"cells":[{"metadata":{},"cell_type":"markdown","source":"# EDA <a id=\"1\"></a>"},{"metadata":{},"cell_type":"markdown","source":"### Install and import necessary packages"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!pip install -q pyicu\n!pip install -q pycld2\n!pip install -q polyglot\n!pip install -q textstat\n!pip install -q googletrans","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os\nimport gc\nimport re\nimport folium\nimport textstat\nfrom scipy import stats\nfrom colorama import Fore, Back, Style, init\n\nimport math\nimport numpy as np\nimport scipy as sp\nimport pandas as pd\n\nimport random\nimport networkx as nx\nfrom pandas import Timestamp\n\nfrom PIL import Image\nfrom IPython.display import SVG\nfrom keras.utils import model_to_dot\n\nimport requests\nfrom IPython.display import HTML\n\nimport seaborn as sns\nfrom tqdm import tqdm\nimport matplotlib.cm as cm\nimport matplotlib.pyplot as plt\n\ntqdm.pandas()\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport transformers\nimport tensorflow as tf\n\nfrom tensorflow.keras.callbacks import Callback\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, CSVLogger\n\nfrom tensorflow.keras.models import Model\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow.keras.optimizers import Adam\nfrom tokenizers import BertWordPieceTokenizer\nfrom tensorflow.keras.layers import Dense, Input, Dropout, Embedding\nfrom tensorflow.keras.layers import LSTM, GRU, Conv1D, SpatialDropout1D\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras import activations\nfrom tensorflow.keras import constraints\nfrom tensorflow.keras import initializers\nfrom tensorflow.keras import regularizers\n\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.optimizers import *\nfrom tensorflow.keras.activations import *\nfrom tensorflow.keras.constraints import *\nfrom tensorflow.keras.initializers import *\nfrom tensorflow.keras.regularizers import *\n\nfrom sklearn import metrics\nfrom sklearn.utils import shuffle\nfrom gensim.models import Word2Vec\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, HashingVectorizer\n\nfrom nltk.stem.wordnet import WordNetLemmatizer \nfrom nltk.tokenize import word_tokenize\nfrom nltk.tokenize import TweetTokenizer  \n\nimport nltk\nfrom textblob import TextBlob\n\nfrom nltk.corpus import wordnet\nfrom nltk.corpus import stopwords\nfrom googletrans import Translator\nfrom nltk import WordNetLemmatizer\nfrom polyglot.detect import Detector\nfrom nltk.stem import WordNetLemmatizer\nfrom wordcloud import WordCloud, STOPWORDS\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\n\nstopword=set(STOPWORDS)\n\nlem = WordNetLemmatizer()\ntokenizer=TweetTokenizer()\n\nnp.random.seed(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the training, validation, and testing datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = \"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/\"\nos.listdir(DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Wordcloud of all comments"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def nonan(x):\n    if type(x) == str:\n        return x.replace(\"\\n\", \"\")\n    else:\n        return \"\"\n\ntext = ' '.join([nonan(abstract) for abstract in train_data[\"comment_text\"]])\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n                      width=1200, height=1000).generate(text)\nfig = px.imshow(wordcloud)\nfig.update_layout(title_text='Common words in comments')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the wordcloud above, we can see the most common words in the comments. These words include \"article\", \"page\", and \"wikipedia\" among other words. More offensive words like \"f**k\" seem to occur less often, indicating that toxic, insulting comments are seen less frequently than non-toxic comments. "},{"metadata":{},"cell_type":"markdown","source":"## Comment words <a id=\"1.3\"></a>\n\nNow, I will look at the number of words present in the comments."},{"metadata":{},"cell_type":"markdown","source":"### Distribution of comment words"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def new_len(x):\n    if type(x) is str:\n        return len(x.split())\n    else:\n        return 0\n\ntrain_data[\"comment_words\"] = train_data[\"comment_text\"].apply(new_len)\nnums = train_data.query(\"comment_words != 0 and comment_words < 200\").sample(frac=0.1)[\"comment_words\"]\nfig = ff.create_distplot(hist_data=[nums],\n                         group_labels=[\"All comments\"],\n                         colors=[\"coral\"])\n\nfig.update_layout(title_text=\"Comment words\", xaxis_title=\"Comment words\", template=\"simple_white\", showlegend=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the plot above, we can see that the distribution of comment words has a strong rightward (positive) skew with maximum probability denisty occuring at around 13 words. As the number of words increases beyond 13, the frequency reduces sharply."},{"metadata":{},"cell_type":"markdown","source":"## Sentiment and polarity <a id=\"1.4\"></a>\n\nSentiment and polarity are quantities that reflect the emotion and intention behind a sentence. Now, I will look at the sentiment of the comments using the NLTK (natural language toolkit) library."},{"metadata":{},"cell_type":"markdown","source":"<center><img src=\"https://i.imgur.com/LQF5WsC.png\" width=\"800px\"></center>"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def polarity(x):\n    if type(x) == str:\n        return SIA.polarity_scores(x)\n    else:\n        return 1000\n    \nSIA = SentimentIntensityAnalyzer()\ntrain_data[\"polarity\"] = train_data[\"comment_text\"].progress_apply(polarity)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Negative sentiment\n\nNegative sentiment refers to negative or pessimistic emotions. It is a score between 0 and 1; the greater the score, the more negative the abstract is."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neg\"] for pols in train_data[\"polarity\"] if pols[\"neg\"] != 0], marker=dict(\n            color='seagreen')\n    ))\n\nfig.update_layout(xaxis_title=\"Negativity sentiment\", title_text=\"Negativity sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that negative sentiment has a strong rightward (positive) skew, indicating that negativity is usually on the lower side. This suggests that most comments are not toxic or negative. In fact, the most common negativity value is around 0.04. Virtually no comments have a negativity greater than 0.8."},{"metadata":{},"cell_type":"markdown","source":"### Negativity vs. Toxicity"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data[\"negativity\"] = train_data[\"polarity\"].apply(lambda x: x[\"neg\"])\n\nnums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"negativity\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"negativity\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n                         group_labels=[\"Toxic\", \"Non-toxic\"],\n                         colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Negativity vs. Toxicity\", xaxis_title=\"Negativity\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the distribution of negativity for toxic and non-toxic comments above. We can clearly see that toxic comments have a significantly greater negative sentiment than toxic comments (on average). The probability density of negativity peaks at around 0 for non-toxic comments, while the negativity for toxic comments are minimum at this point. This suggests that a comment is very likely to be non-toxic if it has a negativity of 0."},{"metadata":{},"cell_type":"markdown","source":"### Positive sentiment\n\nPositive sentiment refers to positive or optimistic emotions. It is a score between 0 and 1; the greater the score, the more positive the abstract is."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"pos\"] for pols in train_data[\"polarity\"] if pols[\"pos\"] != 0], marker=dict(\n            color='indianred')\n    ))\n\nfig.update_layout(xaxis_title=\"Positivity sentiment\", title_text=\"Positivity sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that positive sentiment has a strong rightward (positive) skew, indicating that positivity is usually on the lower side. This suggests that most comments do not express positivity explicitly. In fact, the most common negativity value is around 0.08. Virtually no comments have a positivity greater than 0.8."},{"metadata":{},"cell_type":"markdown","source":"### Positivity vs. Toxicity"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data[\"positivity\"] = train_data[\"polarity\"].apply(lambda x: x[\"pos\"])\n\nnums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"positivity\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"positivity\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n                         group_labels=[\"Toxic\", \"Non-toxic\"],\n                         colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Positivity vs. Toxicity\", xaxis_title=\"Positivity\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the distribution of positivity for toxic and non-toxic comments above. We can see that both the distributions are very similar, indicating that positivity is not an accurate indicator of toxicity in comments. "},{"metadata":{},"cell_type":"markdown","source":"### Neutrality sentiment\n\nNeutrality sentiment refers to the level of bias or opinion in the text. It is a score between 0 and 1; the greater the score, the more neutral/unbiased the abstract is."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neu\"] for pols in train_data[\"polarity\"] if pols[\"neu\"] != 1], marker=dict(\n            color='dodgerblue')\n    ))\n\nfig.update_layout(xaxis_title=\"Neutrality sentiment\", title_text=\"Neutrality sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that the neutrality sentiment distribution has a strong leftward (negative) skew, which is in constrast to the negativity and positivity sentiment distributions. This indicates that the comments tend to be very neutral and unbiased in general. This also suggests that most comments are not highly opinionated and polarizing, meaning that most comments are non-toxic."},{"metadata":{},"cell_type":"markdown","source":"### Neutrality vs. Toxicity"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data[\"neutrality\"] = train_data[\"polarity\"].apply(lambda x: x[\"neu\"])\n\nnums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"neutrality\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"neutrality\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n                         group_labels=[\"Toxic\", \"Non-toxic\"],\n                         colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Neutrality vs. Toxicity\", xaxis_title=\"Neutrality\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that non-toxic comments tend to have a higher neutrality value than toxic comments on average. The probability density of the non-toxic distribution experiences a sudden jump at 1, and the probability density of the toxic distribution is significantly lower at the same point. This suggests that a comment with neutrality close to 1 is more likely to be non-toxic than toxic."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}