{"cells":[{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Acknowledgements\n\n1. [Jigsaw TPU: DistilBERT with Huggingface and Keras](https://www.kaggle.com/xhlulu/jigsaw-tpu-distilbert-with-huggingface-and-keras) ~ by xhulu\n2. [Jigsaw Multilingual: Quick EDA & TPU Modeling](https://www.kaggle.com/ipythonx/jigsaw-multilingual-quick-eda-tpu-modeling) ~ by Innat\n3. [Stop the S@#! - Toxic Comments EDA](https://www.kaggle.com/jagangupta/stop-the-s-toxic-comments-eda) ~ by Jagan\n4. [Google Translate API (googletrans)](https://pypi.org/project/googletrans/) ~ by ssut\n5. [Polyglot](https://pypi.org/project/polyglot/) ~ by aboSamoor\n6. [Dale–Chall readability formula](https://en.wikipedia.org/wiki/Dale%E2%80%93Chall_readability_formula) ~ by Wikipedia\n7. [Automated readability index](https://en.wikipedia.org/wiki/Automated_readability_index) ~ by Wikipedia\n8. [Flesch–Kincaid readability tests](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests) ~ by Wikipedia\n9. [BERT Explained: State of the art language model for NLP](https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270) ~ by Rani Horev"},{"metadata":{},"cell_type":"markdown","source":"# Contents\n\n* [EDA](#1)\n * [Preparing the ground](#1.1)\n * [Languages](#1.2)\n * [Comment words](#1.3)\n * [Sentiment and polarity](#1.4)\n * [Readability](#1.5)\n * [Targets](#1.6)\n \n \n* [Modeling](#2)\n * [Preparing the ground](#2.1)\n * [Vanilla neural network](#2.2)\n * [Convolutional neural network](#2.3)\n * [LSTM with Attention](#2.4)\n * [Capsule network](#2.5)\n * [DistilBERT](#2.6)\n \n \n* [Takeaways](#3)\n\n\n* [Ending note](#4)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# EDA \n\nFirst, I will visualize the comments in the training data before moving on to the modeling section.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing the ground "},{"metadata":{},"cell_type":"markdown","source":"### Install and import necessary packages"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"!pip install -q pyicu\n!pip install -q pycld2\n!pip install -q polyglot\n!pip install -q textstat\n!pip install -q googletrans","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install textstat","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install googletrans","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install icu\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport os\nimport gc\nimport re\nimport folium\nimport textstat\nfrom scipy import stats\nfrom colorama import Fore, Back, Style, init\n\nimport math\nimport numpy as np\nimport scipy as sp\nimport pandas as pd\n\nimport random\nimport networkx as nx\nfrom pandas import Timestamp\n\nfrom PIL import Image\nfrom IPython.display import SVG\nfrom keras.utils import model_to_dot\n\nimport requests\nfrom IPython.display import HTML\n\nimport seaborn as sns\nfrom tqdm import tqdm\nimport matplotlib.cm as cm\nimport matplotlib.pyplot as plt\n\ntqdm.pandas()\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\n\nimport transformers\nimport tensorflow as tf\n\nfrom tensorflow.keras.callbacks import Callback\nfrom sklearn.metrics import accuracy_score, roc_auc_score\nfrom tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, CSVLogger\n\nfrom tensorflow.keras.models import Model\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow.keras.optimizers import Adam\nfrom tokenizers import BertWordPieceTokenizer\nfrom tensorflow.keras.layers import Dense, Input, Dropout, Embedding\nfrom tensorflow.keras.layers import LSTM, GRU, Conv1D, SpatialDropout1D\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import optimizers\nfrom tensorflow.keras import activations\nfrom tensorflow.keras import constraints\nfrom tensorflow.keras import initializers\nfrom tensorflow.keras import regularizers\n\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.optimizers import *\nfrom tensorflow.keras.activations import *\nfrom tensorflow.keras.constraints import *\nfrom tensorflow.keras.initializers import *\nfrom tensorflow.keras.regularizers import *\n\nfrom sklearn import metrics\nfrom sklearn.utils import shuffle\nfrom gensim.models import Word2Vec\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.feature_extraction.text import TfidfVectorizer,\\\n CountVectorizer,\\\n HashingVectorizer\n\nfrom nltk.stem.wordnet import WordNetLemmatizer \nfrom nltk.tokenize import word_tokenize\nfrom nltk.tokenize import TweetTokenizer \n\nimport nltk\nfrom textblob import TextBlob\n\nfrom nltk.corpus import wordnet\nfrom nltk.corpus import stopwords\nfrom googletrans import Translator\nfrom nltk import WordNetLemmatizer\nfrom nltk.stem import WordNetLemmatizer\nfrom wordcloud import WordCloud, STOPWORDS\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\n\nstopword=set(STOPWORDS)\n\nlem = WordNetLemmatizer()\ntokenizer=TweetTokenizer()\n\nnp.random.seed(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the training, validation, and testing datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = \"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/\"\nos.listdir(DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_PATH = DATA_PATH + \"test.csv\"\nVAL_PATH = DATA_PATH + \"validation.csv\"\nTRAIN_PATH = DATA_PATH + \"jigsaw-toxic-comment-train.csv\"\n\nval_data = pd.read_csv(VAL_PATH)\ntest_data = pd.read_csv(TEST_PATH)\n\ntrain_data = pd.read_csv(TRAIN_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Wordcloud of all comments"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import wordcloud\ndef nonan(x):\n if type(x) == str:\n return x.replace(\"\\n\", \"\")\n else:\n return \"\"\n\ntext = ' '.join([nonan(abstract) for abstract in train_data[\"comment_text\"]])\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n width=1200, height=1000).generate(text)\nfig = px.imshow(wordcloud)\nfig.update_layout(title_text='Common words in comments')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the wordcloud above, we can see the most common words in the comments. These words include \"wikipedia\", \"page\", and \"article\" among other words. More offensive words like \"f**k\" seem to occur less often, indicating that toxic, insulting comments are seen less frequently than non-toxic comments. "},{"metadata":{},"cell_type":"markdown","source":"## Languages \n\nNow, I will analyze the distribution of languages in the dataset. To detect the language of comments in the dataset, I used the **Polyglot** package, which takes text as input and predicts the language of the text.\n\n![](\"https://i.imgur.com/Hoa6IWg.png\")
"},{"metadata":{"trusted":true},"cell_type":"code","source":"import polyglot\nfrom polyglot.detect import Detector","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\ndef get_language(text):\n return Detector(\"\".join(x for x in text if x.isprintable()), quiet=True).languages[0].name\n\ntrain_data[\"lang\"] = train_data[\"comment_text\"].progress_apply(get_language)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### English vs. Non-English"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"lang_list = sorted(list(set(train_data[\"lang\"])))\ncounts = [list(train_data[\"lang\"]).count(cont) for cont in lang_list]\ndf = pd.DataFrame(np.transpose([lang_list, counts]))\ndf.columns = [\"Language\", \"Count\"]\ndf[\"Count\"] = df[\"Count\"].apply(int)\n\ndf_en = pd.DataFrame(np.transpose([[\"English\", \"Non-English\"], [max(counts), sum(counts) - max(counts)]]))\ndf_en.columns = [\"Language\", \"Count\"]\n\nfig = px.bar(df_en, x=\"Language\", y=\"Count\", title=\"Language of comments\", color=\"Language\", text=\"Count\")\nfig.update_layout(template=\"plotly_white\")\nfig.data[0].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[0].marker.line.width = 0.5\nfig.data[1].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[1].marker.line.width = 0.5\nfig.data[0].textfont.color = \"black\"\nfig.data[0].textposition = \"outside\"\nfig.data[1].textfont.color = \"black\"\nfig.data[1].textposition = \"outside\"\nfig","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that English comments dominate the training data, with a total of 220636 comments written in English and a mere 2913 comments written in languages other than English. There is a heavy imbalance in the language of comments in the training data."},{"metadata":{},"cell_type":"markdown","source":"### Bar chart of non-English languages"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(df.query(\"Language != 'English' and Language != 'un'\").query(\"Count >= 50\"),\n y=\"Language\", x=\"Count\", title=\"Language of non-English comments\", template=\"plotly_white\", color=\"Language\", text=\"Count\", orientation=\"h\")\nfig.update_traces(marker=dict(line=dict(width=0.75,\n color='black')), textposition=\"outside\")\nfig.update_layout(showlegend=False)\nfig","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that German, Scots, and Danish are the most common non-English languages featuring in the dataset, with more than 100 comments in each language. Spanish, Persian, and Arabic are not far behind. We can thus conclude that Europe and the middle-east are the most represented regions in the dataset."},{"metadata":{},"cell_type":"markdown","source":"### Pie chart of non-English languages"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure([go.Pie(labels=df.query(\"Language != 'English' and Language != 'un'\").query(\"Count >= 50\")[\"Language\"],\n values=df.query(\"Language != 'English' and Language != 'un'\").query(\"Count >= 50\")[\"Count\"])])\nfig.update_layout(title_text=\"Pie chart of non-English languages\", template=\"plotly_white\")\nfig.data[0].marker.colors = [px.colors.qualitative.Plotly[2:]]\nfig.data[0].textfont.color = \"black\"\nfig.data[0].textposition = \"outside\"\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the pie chart above, we can once again see that German, Danish, and Scots with more than 15% of the pie belonging to each of these three languages."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_country(language):\n if language == \"German\":\n return \"Germany\"\n if language == \"Scots\":\n return \"Scotland\"\n if language == \"Danish\":\n return \"Denmark\"\n if language == \"Arabic\":\n return \"Saudi Arabia\"\n if language == \"Spanish\":\n return \"Spain\"\n if language == \"Persian\":\n return \"Iran\"\n if language == \"Greek\":\n return \"Greece\"\n if language == \"Portuguese\":\n return \"Portugal\"\n if language == \"English\":\n return \"United Kingdom\"\n if language == \"Hindi\":\n return \"India\"\n if language == \"Albanian\":\n return \"Albania\"\n if language == \"Bosnian\":\n return \"Bosnia and Herzegovina\"\n if language == \"Croatian\":\n return \"Croatia\"\n if language == \"Dutch\":\n return \"Netherlands\"\n if language == \"Russian\":\n return \"Russia\"\n if language == \"Vietnamese\":\n return \"Vietnam\"\n if language == \"Somali\":\n return \"Somalia\"\n if language == \"Turkish\":\n return \"Turkey\"\n if language == \"Serbian\":\n return \"Serbia\"\n if language == \"Indonesian\":\n return \"Indonesia\"\n if language == \"Manx\":\n return \"Ireland\"\n if language == \"Scots\":\n return \"Scotland\"\n if language == \"Latin\":\n return \"Holy See (Vatican City State)\"\n if language == \"Afrikaans\":\n return \"South Africa\"\n return \"None\"\n \ndf[\"country\"] = df[\"Language\"].progress_apply(get_country)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### World plot of non-English languages"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'English' and Language != 'un' and country != 'None'\").query(\"Count >= 5\"), locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Countries of non-English languages\", color=\"Count\",\n template=\"plotly\", color_continuous_scale=\"agsunset\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the word plot above, we can see that western Europe and the middle-east are the most represented regions in the dataset. Africa, Asia, and eastern Europe are relatively under-represented."},{"metadata":{},"cell_type":"markdown","source":"### Non-English European "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'English' and Language != 'un' and country != 'None'\"), locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Non-English European countries\", color=\"Count\",\n template=\"plotly\", color_continuous_scale=\"aggrnyl\", scope=\"europe\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that German and English are the most common European languages to feature in the dataset, although Spanish and Greek are not far behind."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'English' and Language != 'un' and country != 'None'\"), locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Asian countries\", color=\"Count\",\n template=\"plotly\", color_continuous_scale=\"spectral\", scope=\"asia\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This plot shows that middle-eastern languages such as Arabic and Persian are represented more than languages from the Indian subcontinent or south-east Asia, such as Hindi, Vietnamese, and Indonesian. There is not a single comment in Mandarin, Korean, or Japanese!"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.choropleth(df.query(\"Language != 'English' and Language != 'un' and country != 'None'\").query(\"Count >= 5\"), locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"African countries\", color=\"Count\",\n template=\"plotly\", color_continuous_scale=\"agsunset\", scope=\"africa\")\n# fig.data[0].marker.line.color = 'rgb(0, 0, 0)'\n# fig.data[0].marker.line.width = 0.2\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that Africa is not as well represented as the other continents in the dataset. The two most common African languages in the dataset are Afrikaans and Somali."},{"metadata":{},"cell_type":"markdown","source":"## Comment words \n\nNow, I will look at the number of words present in the comments."},{"metadata":{},"cell_type":"markdown","source":"### Distribution of comment words"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def new_len(x):\n if type(x) is str:\n return len(x.split())\n else:\n return 0\n\ntrain_data[\"comment_words\"] = train_data[\"comment_text\"].apply(new_len)\nnums = train_data.query(\"comment_words != 0 and comment_words < 200\").sample(frac=0.1)[\"comment_words\"]\nfig = ff.create_distplot(hist_data=[nums],\n group_labels=[\"All comments\"],\n colors=[\"coral\"])\n\nfig.update_layout(title_text=\"Comment words\", xaxis_title=\"Comment words\", template=\"simple_white\", showlegend=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the plot above, we can see that the distribution of comment words has a strong rightward (positive) skew with maximum probability denisty occuring at around 13 words. As the number of words increases beyond 13, the frequency reduces sharply."},{"metadata":{},"cell_type":"markdown","source":"### Average comment words vs. Language"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"comment_words\"]]))\ndf.columns = [\"Language\", \"Average_comment_words\"]\ndf[\"Average_comment_words\"] = df[\"Average_comment_words\"].apply(float)\ndf = df.query(\"Average_comment_words < 500\")\nfig = go.Figure(go.Bar(x=df[\"Language\"], y=df[\"Average_comment_words\"]))\n\nfig.update_layout(xaxis_title=\"Language\", yaxis_title=\"Average comment words\", title_text=\"Average comment words vs. language\", template=\"plotly_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the average comment words in each language above. Certain languages tend to have more words on average than other languages. For example, comments written in Akan, Persian, and Sinhala have more than 300 words on average! This may be due to the small number of samples in these languages and presence of one or two outliers."},{"metadata":{},"cell_type":"markdown","source":"### Average comment length vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average comment length vs. Country\", color=\"Average_comment_words\",\n template=\"plotly\", color_continuous_scale=\"aggrnyl\")\nfig","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot above, we can see that certain regions in the world tend to have a higher average comment length comment length than other countries. Persian, Arabic, and Hindi comments all have at least 100 words on average! Most long-comment languages seem to originate in Asia."},{"metadata":{},"cell_type":"markdown","source":"## Sentiment and polarity \n\nSentiment and polarity are quantities that reflect the emotion and intention behind a sentence. Now, I will look at the sentiment of the comments using the NLTK (natural language toolkit) library."},{"metadata":{},"cell_type":"markdown","source":"![](\"https://i.imgur.com/LQF5WsC.png\")
"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def polarity(x):\n if type(x) == str:\n return SIA.polarity_scores(x)\n else:\n return 1000\n \nSIA = SentimentIntensityAnalyzer()\ntrain_data[\"polarity\"] = train_data[\"comment_text\"].progress_apply(polarity)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Negative sentiment\n\nNegative sentiment refers to negative or pessimistic emotions. It is a score between 0 and 1; the greater the score, the more negative the abstract is."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neg\"] for pols in train_data[\"polarity\"] if pols[\"neg\"] != 0], marker=dict(\n color='seagreen')\n ))\n\nfig.update_layout(xaxis_title=\"Negativity sentiment\", title_text=\"Negativity sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that negative sentiment has a strong rightward (positive) skew, indicating that negativity is usually on the lower side. This suggests that most comments are not toxic or negative. In fact, the most common negativity value is around 0.04. Virtually no comments have a negativity greater than 0.8."},{"metadata":{},"cell_type":"markdown","source":"### Negativity vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data[\"negativity\"] = train_data[\"polarity\"].apply(lambda x: x[\"neg\"])\ndf = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"negativity\"].tolist()]))\ndf.columns = [\"Language\", \"Negativity\"]\ndf[\"Negativity\"] = df[\"Negativity\"].apply(float)\ndf = df.query(\"Negativity != 0\")\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average negative sentiment vs. Country\", color=\"Negativity\",\n template=\"plotly\", color_continuous_scale=\"greens\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot, we can see that the language with highest average negativity is Afrikaans. Also, languages from western Europe and south-east Asia tend to have higher toxicity than Hindi and Russian."},{"metadata":{},"cell_type":"markdown","source":"### Negativity vs. Toxicity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"negativity\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"negativity\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Negativity vs. Toxicity\", xaxis_title=\"Negativity\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the distribution of negativity for toxic and non-toxic comments above. We can clearly see that toxic comments have a significantly greater negative sentiment than toxic comments (on average). The probability density of negativity peaks at around 0 for non-toxic comments, while the negativity for toxic comments are minimum at this point. This suggests that a comment is very likely to be non-toxic if it has a negativity of 0."},{"metadata":{},"cell_type":"markdown","source":"### Positive sentiment\n\nPositive sentiment refers to positive or optimistic emotions. It is a score between 0 and 1; the greater the score, the more positive the abstract is."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"pos\"] for pols in train_data[\"polarity\"] if pols[\"pos\"] != 0], marker=dict(\n color='indianred')\n ))\n\nfig.update_layout(xaxis_title=\"Positivity sentiment\", title_text=\"Positivity sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that positive sentiment has a strong rightward (positive) skew, indicating that positivity is usually on the lower side. This suggests that most comments do not express positivity explicitly. In fact, the most common negativity value is around 0.08. Virtually no comments have a positivity greater than 0.8."},{"metadata":{},"cell_type":"markdown","source":"### Positivity vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data[\"positivity\"] = train_data[\"polarity\"].apply(lambda x: x[\"pos\"])\ndf = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"positivity\"].tolist()]))\ndf.columns = [\"Language\", \"Positivity\"]\ndf[\"Positivity\"] = df[\"Positivity\"].apply(float)\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average positive sentiment vs. Country\", color=\"Positivity\",\n template=\"plotly\", color_continuous_scale=\"reds\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot, we can see that the languages with the highest average positivity are English, Spanish, Portuguese, and Danish."},{"metadata":{},"cell_type":"markdown","source":"### Positivity vs. Toxicity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"positivity\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"positivity\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Positivity vs. Toxicity\", xaxis_title=\"Positivity\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the distribution of positivity for toxic and non-toxic comments above. We can see that both the distributions are very similar, indicating that positivity is not an accurate indicator of toxicity in comments. "},{"metadata":{},"cell_type":"markdown","source":"### Neutrality sentiment\n\nNeutrality sentiment refers to the level of bias or opinion in the text. It is a score between 0 and 1; the greater the score, the more neutral/unbiased the abstract is."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"neu\"] for pols in train_data[\"polarity\"] if pols[\"neu\"] != 1], marker=dict(\n color='dodgerblue')\n ))\n\nfig.update_layout(xaxis_title=\"Neutrality sentiment\", title_text=\"Neutrality sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above plot, we can see that the neutrality sentiment distribution has a strong leftward (negative) skew, which is in constrast to the negativity and positivity sentiment distributions. This indicates that the comments tend to be very neutral and unbiased in general. This also suggests that most comments are not highly opinionated and polarizing, meaning that most comments are non-toxic."},{"metadata":{},"cell_type":"markdown","source":"### Neutrality vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data[\"neutrality\"] = train_data[\"polarity\"].apply(lambda x: x[\"neu\"])\ndf = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"neutrality\"].tolist()]))\ndf.columns = [\"Language\", \"Neutrality\"]\ndf[\"Neutrality\"] = df[\"Neutrality\"].apply(float)\ndf = df.query(\"Neutrality != 1\")\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average neutral sentiment vs. Country\", color=\"Neutrality\",\n template=\"plotly\", color_continuous_scale=\"blues\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot, we can see that the languages with the highest neutrality are Persian, Hindi, and Russian. Few western European languages like German and English seem to have a lower average neutrality than most other languages."},{"metadata":{},"cell_type":"markdown","source":"### Neutrality vs. Toxicity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"neutrality\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"neutrality\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Neutrality vs. Toxicity\", xaxis_title=\"Neutrality\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that non-toxic comments tend to have a higher neutrality value than toxic comments on average. The probability density of the non-toxic distribution experiences a sudden jump at 1, and the probability density of the toxic distribution is significantly lower at the same point. This suggests that a comment with neutrality close to 1 is more likely to be non-toxic than toxic."},{"metadata":{},"cell_type":"markdown","source":"### Compound sentiment\n\nCompoundness sentiment refers to the total level of sentiment in the sentence. It is a score between -1 and 1; the greater the score, the more emotional the abstract is."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=[pols[\"compound\"] for pols in train_data[\"polarity\"] if pols[\"compound\"] != 0], marker=dict(\n color='orchid')\n ))\n\nfig.update_layout(xaxis_title=\"Compound sentiment\", title_text=\"Compound sentiment\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the distribution above, we can see that compound sentiment is evenly distributed across the specturm (from -1 to 1) with very high variance and random peaks throughout the range."},{"metadata":{},"cell_type":"markdown","source":"### Average compound sentiment vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_data[\"compound\"] = train_data[\"polarity\"].apply(lambda x: x[\"compound\"])\ndf = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"compound\"].tolist()]))\ndf.columns = [\"Language\", \"Compound\"]\ndf[\"Compound\"] = df[\"Compound\"].apply(float)\ndf = df.query(\"Compound != 0\")\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average compound sentiment vs. Country\", color=\"Compound\",\n template=\"plotly\", color_continuous_scale=\"purples\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot above, we can see that western European countries, south-east Asia, and Turkey have a lower average compound sentiment than most other countries. India, Russia, and Iran are among the countries with the maximum compound sentiment."},{"metadata":{},"cell_type":"markdown","source":"### Compound sentiment vs. Toxicity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"compound\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"compound\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Compound vs. Toxicity\", xaxis_title=\"Compound\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that compound sentiment tends to be higher for non-toxic comments as compared to toxic comments. The non-toxic distribution has a leftward (negative) skew, while the toxic distribution has a positive (rightward) skew. This indicates that non-toxic comments tend to have a higher compound sentiment than toxic comments on average."},{"metadata":{},"cell_type":"markdown","source":"## Readability \n\nReadability is an indication of how \"easy\" it is to read some text. There are several metrics that can be used to measure the readability of a piece of text, including Flesch reading ease, automated readability, and Dale-Chall readability. I will be using the textstat library to calculate the readability of comments in the dataset.\n\n![](\"https://i.imgur.com/ySwAGNi.png\")
"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_data[\"flesch_reading_ease\"] = train_data[\"comment_text\"].progress_apply(textstat.flesch_reading_ease)\ntrain_data[\"automated_readability\"] = train_data[\"comment_text\"].progress_apply(textstat.automated_readability_index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data[\"dale_chall_readability\"] = train_data[\"comment_text\"].progress_apply(textstat.dale_chall_readability_score)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Flesch reading ease\n\nThe Flesch readability ease is an indicator designed to quantify how difficult a passage is to understand. The Flesch readability ease is calculated mathematically using the formula below:\n\n![](\"https://i.imgur.com/nead3Kh.png\")
"},{"metadata":{},"cell_type":"markdown","source":"From the above formula, we can see that comments with shorter words and fewer words per sentence are more \"readable\"."},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Flesch reading ease"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=train_data.query(\"flesch_reading_ease > 0\")[\"flesch_reading_ease\"], marker=dict(\n color='darkorange')\n ))\n\nfig.update_layout(xaxis_title=\"Flesch reading ease\", title_text=\"Flesch reading ease\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Flesch reading ease distribution has a slight leftward (negative) skew, although the distribution is roughly normal. The most common values of the metric lie between 66 and 66.5."},{"metadata":{},"cell_type":"markdown","source":"### Flesch reading ease vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"flesch_reading_ease\"].tolist()]))\ndf.columns = [\"Language\", \"flesch_reading_ease\"]\ndf[\"flesch_reading_ease\"] = df[\"flesch_reading_ease\"].apply(float)\ndf = df.query(\"flesch_reading_ease > 0\")\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average Flesch reading ease vs. Country\", color=\"flesch_reading_ease\",\n template=\"plotly\", color_continuous_scale=\"oranges\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot above, we can see that the Flesch readability ease is maximum in the Russian and Vietnamese languages. These languages have few words per sentence and few syllables per word, indicating that they are \"easier\" to read."},{"metadata":{},"cell_type":"markdown","source":"### Flesch reading ease vs. Toxicity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"flesch_reading_ease\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"flesch_reading_ease\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Flesch reading ease vs. Toxicity\", xaxis_title=\"Flesch reading ease\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the distribution of Flesch reading ease for toxic and non-toxic comments above. We can see that both the distributions are very similar, indicating that Flesch reading ease is not an accurate indicator of toxicity in comments. "},{"metadata":{},"cell_type":"markdown","source":"### Automated readability\n\nThe automated readability index (ARI) is a readability test, designed to gauge the understandability of a text. The automated readability index is calculated mathematically using the formula below:\n\n![](\"https://i.imgur.com/7RukUx5.png\")
"},{"metadata":{},"cell_type":"markdown","source":"From the above formula, we can see that comments with shorter words and fewer words per sentence are more \"readable\"."},{"metadata":{},"cell_type":"markdown","source":"### Distribution of automated readability"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=train_data.query(\"automated_readability < 100\")[\"automated_readability\"], marker=dict(\n color='mediumaquamarine')\n ))\n\nfig.update_layout(xaxis_title=\"Automated readability\", title_text=\"Automated readability\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The automated readability distribution has a slight rightward (positive) skew, although the distribution has a roughly normal shape. The most common value of automated readability in the dataset is approximately 9. Very few comments have a readability ease greater than 60."},{"metadata":{},"cell_type":"markdown","source":"### Automated readability vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"automated_readability\"].tolist()]))\ndf.columns = [\"Language\", \"automated_readability\"]\ndf[\"automated_readability\"] = df[\"automated_readability\"].apply(float)\ndf = df.query(\"automated_readability < 100\")\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Automated readability vs. Country\", color=\"automated_readability\",\n template=\"plotly\", color_continuous_scale=\"GnBu\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the world plot above, we can see that automated readability is maximum in Hindi, Arabic, and Somali comments. Whereas, Turkish, English, and south-east Asian comments seem to have relatively lower automated readability value than most countries."},{"metadata":{},"cell_type":"markdown","source":"### Automated readability vs. Toxicity"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"automated_readability\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"automated_readability\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Automated readability vs. Toxicity\", xaxis_title=\"Automated readability\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have plotted the distribution of automated readability for toxic and non-toxic comments above. We can see that both the distributions are very similar, indicating that automated readability is not an accurate indicator of toxicity in comments. "},{"metadata":{},"cell_type":"markdown","source":"### Dale-Chall readability\n\nThe Dale–Chall readability formula is a readability test that provides a numeric gauge of the comprehension difficulty that readers come upon when reading a text. It uses a list of 3000 words that groups of fourth-grade American students could reliably understand, considering any word not on that list to be difficult. The Dale-Chall readability is calculated mathematically using the formula below:\n\n![](\"https://i.imgur.com/KFG0QuU.png\")
"},{"metadata":{},"cell_type":"markdown","source":"From the above formula, we can see that comments with fewer \"difficult\" words has a better readability and therefore a lower Dale-Chall readability score."},{"metadata":{},"cell_type":"markdown","source":"### Distribution of Dale-Chall readability"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(go.Histogram(x=train_data.query(\"dale_chall_readability < 20\")[\"dale_chall_readability\"], marker=dict(\n color='deeppink')\n ))\n\nfig.update_layout(xaxis_title=\"Dale-Chall readability\", title_text=\"Dale-Chall readability\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Dale-Chall readability distribution has a slight rightward (positive) skew, although the distribution has a roughly normal shape. There are also peaks to the left of the main distribution. The most common value of automated readability in the dataset is approximately 7. Very few comments have a readability ease greater than 20."},{"metadata":{},"cell_type":"markdown","source":"### Dale-Chall readability vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"dale_chall_readability\"].tolist()]))\ndf.columns = [\"Language\", \"dale_chall_readability\"]\ndf[\"dale_chall_readability\"] = df[\"dale_chall_readability\"].apply(float)\ndf = df.query(\"dale_chall_readability < 20\")\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Dale-Chall readability vs. Country\", color=\"dale_chall_readability\",\n template=\"plotly\", color_continuous_scale=\"PuRd\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Dale-Chall readability score seems to be maximum in middle-eastern and south-east Asian languages. Russian and Arabic, on the other hand, have a lower Dale-Chall readability than most other languages in the dataset."},{"metadata":{},"cell_type":"markdown","source":"### Dale-Chall readability"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nums_1 = train_data.sample(frac=0.1).query(\"toxic == 1\")[\"dale_chall_readability\"]\nnums_2 = train_data.sample(frac=0.1).query(\"toxic == 0\")[\"dale_chall_readability\"]\n\nfig = ff.create_distplot(hist_data=[nums_1, nums_2],\n group_labels=[\"Toxic\", \"Non-toxic\"],\n colors=[\"darkorange\", \"dodgerblue\"], show_hist=False)\n\nfig.update_layout(title_text=\"Dale-Chall readability vs. Toxicity\", xaxis_title=\"Dale-Chall readability\", template=\"simple_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Dale-Chall readability seems to be higher (on average) for non-toxic comments, indicating that non-toxic comments use more \"sophisticated\" or \"difficult\" language. Toxic comments, on the other hand, are more blunt. We can see this from the fact that the non-toxic distribution peaks at a higher value than the toxic distribution."},{"metadata":{},"cell_type":"markdown","source":"## Targets \n\nNow, I will visualize the targets in the dataset."},{"metadata":{},"cell_type":"markdown","source":"### Wordclouds for different categories"},{"metadata":{},"cell_type":"markdown","source":"### Non-toxic vs. Toxic"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"clean_mask=np.array(Image.open(\"../input/imagesforkernal/safe-zone.png\"))\nclean_mask=clean_mask[:,:,1]\n\nsubset = train_data.query(\"toxic == 0\")\ntext = subset.comment_text.values\nwc = WordCloud(background_color=\"black\",max_words=2000,mask=clean_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(7.5, 7.5))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Clean Comments\", fontsize=16)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)\nplt.show()\n\nclean_mask=np.array(Image.open(\"../input/imagesforkernal/swords.png\"))\nclean_mask=clean_mask[:,:,1]\n\nsubset = train_data.query(\"toxic == 1\")\ntext = subset.comment_text.values\nwc = WordCloud(background_color=\"black\",max_words=2000,mask=clean_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(7.5, 7.5))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Toxic Comments\", fontsize=16)\nplt.imshow(wc.recolor(colormap= 'viridis' , random_state=17), alpha=0.98)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the above wordclouds, that toxic comments use more insluting or hateful words such as \"f**k\", while the non-toxic comments do not usually use such words."},{"metadata":{},"cell_type":"markdown","source":"### Obscene vs. Severe Toxic vs. Threat vs. Insult"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"toxic_mask=np.array(Image.open(\"../input/imagesforkernal/toxic-sign.png\"))\ntoxic_mask=toxic_mask[:,:,1]\n#wordcloud for clean comments\nsubset=train_data.query(\"obscene == 1\")\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=4000,mask=toxic_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.figure(figsize=(20,20))\nplt.subplot(221)\nplt.axis(\"off\")\nplt.title(\"Words frequented in Obscene Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'gist_earth' , random_state=244), alpha=0.98)\n\n#Severely toxic comments\nplt.subplot(222)\nsevere_toxic_mask=np.array(Image.open(\"../input/imagesforkernal/bomb.png\"))\nsevere_toxic_mask=severe_toxic_mask[:,:,1]\nsubset=train_data[train_data.severe_toxic==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=severe_toxic_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Severe Toxic Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Reds' , random_state=244), alpha=0.98)\n\n#Threat comments\nplt.subplot(223)\nthreat_mask=np.array(Image.open(\"../input/imagesforkernal/anger.png\"))\nthreat_mask=threat_mask[:,:,1]\nsubset=train_data[train_data.threat==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=threat_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in Threatening Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'summer' , random_state=2534), alpha=0.98)\n\n#insult\nplt.subplot(224)\ninsult_mask=np.array(Image.open(\"../input/imagesforkernal/swords.png\"))\ninsult_mask=insult_mask[:,:,1]\nsubset=train_data[train_data.insult==1]\ntext=subset.comment_text.values\nwc= WordCloud(background_color=\"black\",max_words=2000,mask=insult_mask,stopwords=stopword)\nwc.generate(\" \".join(text))\nplt.axis(\"off\")\nplt.title(\"Words frequented in insult Comments\", fontsize=20)\nplt.imshow(wc.recolor(colormap= 'Paired_r' , random_state=244), alpha=0.98)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the above wordclouds, we can see that most of these categories use insulting/hateful language. But, the threat category seems to be slightly different from the remaining categories, as it uses words like \"kill\" and \"die\", indicating that most threats involve threats to kill someone."},{"metadata":{},"cell_type":"markdown","source":"### Pie chart of targets"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(data=[\n go.Pie(labels=train_data.columns[2:7],\n values=train_data.iloc[:, 2:7].sum().values, marker=dict(colors=px.colors.qualitative.Plotly))\n])\nfig.update_traces(textposition='outside', textfont=dict(color=\"black\"))\nfig.update_layout(title_text=\"Pie chart of labels\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the pie chart above, we can see that the most common target is toxic, and the other targets, such as insult and threat are relatively uncommon."},{"metadata":{},"cell_type":"markdown","source":"### Bar chart of targets"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = go.Figure(data=[\n go.Bar(y=train_data.columns[2:7],\n x=train_data.iloc[:, 2:7].sum().values, marker=dict(color=px.colors.qualitative.Plotly))\n])\n\nfig.data[0].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[0].marker.line.width = 0.75\nfig.update_traces(orientation=\"h\")\nfig.update_layout(title_text=\"Bar chart of labels\", template=\"plotly_white\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the bar chart above, we can once again see that toxic is the most common target, while threat is the rarest."},{"metadata":{},"cell_type":"markdown","source":"### Toxicity vs. Country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = pd.DataFrame(np.transpose([lang_list, train_data.groupby(\"lang\").mean()[\"toxic\"].tolist()]))\ndf.columns = [\"Language\", \"toxicity\"]\ndf[\"toxicity\"] = df[\"toxicity\"].apply(float)\ndf[\"country\"] = df[\"Language\"].apply(get_country)\ndf = df.query(\"country != 'None'\")\n\nfig = px.choropleth(df, locations=\"country\", hover_name=\"country\",\n projection=\"natural earth\", locationmode=\"country names\", title=\"Average toxicity vs. Country\", color=\"toxicity\",\n template=\"plotly\", color_continuous_scale=\"tealrose\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above world plot, we can see that Irish and Afrikaans are the countries with maximum average toxicity, while most other countries have a similar average toxicity value."},{"metadata":{},"cell_type":"markdown","source":"# Modeling \n\nNow, I will show how different deep learning models can be used to classify toxic comments."},{"metadata":{},"cell_type":"markdown","source":"## Preparing the ground "},{"metadata":{},"cell_type":"markdown","source":"### Clean the text (remove usernames and links)"},{"metadata":{"trusted":true},"cell_type":"code","source":"val = val_data\ntrain = train_data\n\ndef clean(text):\n text = text.fillna(\"fillna\").str.lower()\n text = text.map(lambda x: re.sub('\\\\n',' ',str(x)))\n text = text.map(lambda x: re.sub(\"\\[\\[User.*\",'',str(x)))\n text = text.map(lambda x: re.sub(\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\",'',str(x)))\n text = text.map(lambda x: re.sub(\"\\(http://.*?\\s\\(http://.*\\)\",'',str(x)))\n return text\n\nval[\"comment_text\"] = clean(val[\"comment_text\"])\ntest_data[\"content\"] = clean(test_data[\"content\"])\ntrain[\"comment_text\"] = clean(train[\"comment_text\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define ROC-AUC evaluation metric"},{"metadata":{"trusted":true},"cell_type":"code","source":"class RocAucEvaluation(Callback):\n def __init__(self, validation_data=(), interval=1):\n super(Callback, self).__init__()\n\n self.interval = interval\n self.X_val, self.y_val = validation_data\n\n def on_epoch_end(self, epoch, logs={}):\n if epoch % self.interval == 0:\n y_pred = self.model.predict(self.X_val, verbose=0)\n score = roc_auc_score(self.y_val, y_pred)\n print(\"\\n ROC-AUC - epoch: {:d} - score: {:.6f}\".format(epoch+1, score))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define function to tokenize (encode) comments"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fast_encode(texts, tokenizer, chunk_size=240, maxlen=512):\n tokenizer.enable_truncation(max_length=maxlen)\n tokenizer.enable_padding(max_length=maxlen)\n all_ids = []\n \n for i in range(0, len(texts), chunk_size):\n text_chunk = texts[i:i+chunk_size].tolist()\n encs = tokenizer.encode_batch(text_chunk)\n all_ids.extend([enc.ids for enc in encs])\n \n return np.array(all_ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Setup TPU configuration"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nAUTO = tf.data.experimental.AUTOTUNE\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(tpu)\ntf.tpu.experimental.initialize_tpu_system(tpu)\nstrategy = tf.distribute.experimental.TPUStrategy(tpu)\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('jigsaw-multilingual-toxic-comment-classification')\n\nEPOCHS = 2\nBATCH_SIZE = 32 * strategy.num_replicas_in_sync","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load BERT tokenizer"},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenizer = transformers.DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')\n\nsave_path = '/kaggle/working/distilbert_base_uncased/'\nif not os.path.exists(save_path):\n os.makedirs(save_path)\ntokenizer.save_pretrained(save_path)\n\nfast_tokenizer = BertWordPieceTokenizer('distilbert_base_uncased/vocab.txt', \n lowercase=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Encode comments and get targets"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = fast_encode(train.comment_text.astype(str), \n fast_tokenizer, maxlen=512)\nx_valid = fast_encode(val_data.comment_text.astype(str).values, \n fast_tokenizer, maxlen=512)\nx_test = fast_encode(test_data.content.astype(str).values, \n fast_tokenizer, maxlen=512)\n\ny_valid = val.toxic.values\ny_train = train.toxic.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define training, validation, and testing datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = (\n tf.data.Dataset\n .from_tensor_slices((x_train, y_train))\n .repeat()\n .shuffle(2048)\n .batch(BATCH_SIZE)\n .prefetch(AUTO)\n)\n\nvalid_dataset = (\n tf.data.Dataset\n .from_tensor_slices((x_valid, y_valid))\n .batch(BATCH_SIZE)\n .cache()\n .prefetch(AUTO)\n)\n\ntest_dataset = (\n tf.data.Dataset\n .from_tensor_slices(x_test)\n .batch(BATCH_SIZE)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Vanilla neural network \n\nVanilla neural network refers to the classic neural network architecture."},{"metadata":{},"cell_type":"markdown","source":"![](\"https://i.imgur.com/ReZ9Ppl.png\")
"},{"metadata":{},"cell_type":"markdown","source":"Vanilla neural networks consist of sequential layers that perform simple matrix multiplications and vector additions, until we reach the output layer. The propagation of values in a VNN can be represented with the following equation:\n\n![](\"https://i.imgur.com/xbtn9ex.png\")
\n\nwhere *W* is the weight matrix and *b* is the bias vector in layer *n*."},{"metadata":{},"cell_type":"markdown","source":"I will use the pretrained BERT embeddings as input, add the word vectors, and pass it through a VNN, as though it was tabular data and get the probability of the comment being toxic. The approach can be summarized using the flowchart below:\n\n![](\"https://i.imgur.com/ORDcivv.png\")
"},{"metadata":{},"cell_type":"markdown","source":"The same logic is implemented below using tf.keras."},{"metadata":{},"cell_type":"markdown","source":"### Define VNN model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_vnn_model(transformer, max_len):\n input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n \n embed = transformer.weights[0].numpy()\n embedding = Embedding(np.shape(embed)[0], np.shape(embed)[1],\n input_length=max_len, weights=[embed],\n trainable=False)(input_word_ids)\n \n conc = K.sum(embedding, axis=2)\n conc = Dense(128, activation='relu')(conc)\n conc = Dense(1, activation='sigmoid')(conc)\n \n model = Model(inputs=input_word_ids, outputs=conc)\n \n model.compile(Adam(lr=0.01), \n loss='binary_crossentropy', \n metrics=['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build model and check summary"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"with strategy.scope():\n transformer_layer = transformers.TFDistilBertModel.\\\n from_pretrained('distilbert-base-multilingual-cased')\n model_vnn = build_vnn_model(transformer_layer, max_len=512)\n\nmodel_vnn.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize network architecture"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model_vnn, dpi=70).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define ReduceLROnPlateau callback"},{"metadata":{"trusted":true},"cell_type":"code","source":"def callback():\n cb = []\n\n reduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', \n factor=0.3, patience=3, \n verbose=1, mode='auto', \n epsilon=0.0001, cooldown=1, min_lr=0.000001)\n cb.append(reduceLROnPlat)\n log = CSVLogger('log.csv')\n cb.append(log)\n\n RocAuc = RocAucEvaluation(validation_data=(x_valid, y_valid), interval=1)\n cb.append(RocAuc)\n \n return cb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"N_STEPS = x_train.shape[0] // BATCH_SIZE\ncalls = callback()\n\ntrain_history = model_vnn.fit(\n train_dataset,\n steps_per_epoch=N_STEPS,\n validation_data=valid_dataset,\n callbacks = calls,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model predictions\n\nNow, I will visualize the performance of the model on few validation samples."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"translator = Translator()\n\ndef visualize_model_preds(model, indices=[0, 17, 1, 24]):\n comments = val_data.comment_text.loc[indices].values.tolist()\n preds = model.predict(x_valid[indices].reshape(len(indices), -1))\n\n for idx, i in enumerate(indices):\n if y_valid[i] == 0:\n label = \"Non-toxic\"\n color = f'{Fore.GREEN}'\n symbol = '\\u2714'\n else:\n label = \"Toxic\"\n color = f'{Fore.RED}'\n symbol = '\\u2716'\n\n print('{}{} {}'.format(color, str(idx+1) + \". \" + label, symbol))\n print(f'{Style.RESET_ALL}')\n print(\"ORIGINAL\")\n print(comments[idx]); print(\"\")\n print(\"TRANSLATED\")\n print(translator.translate(comments[idx]).text)\n fig = go.Figure()\n if list.index(sorted(preds[:, 0]), preds[idx][0]) > 1:\n yl = [preds[idx][0], 1 - preds[idx][0]]\n else:\n yl = [1 - preds[idx][0], preds[idx][0]]\n fig.add_trace(go.Bar(x=['Non-Toxic', 'Toxic'], y=yl, marker=dict(color=[\"seagreen\", \"indianred\"])))\n fig.update_traces(name=comments[idx])\n fig.update_layout(xaxis_title=\"Labels\", yaxis_title=\"Probability\", template=\"plotly_white\", title_text=\"Predictions for validation comment #{}\".format(idx+1))\n fig.show()\n \nvisualize_model_preds(model_vnn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the model gets only two out of four answers correct. We need to find a better model to achieve a higher accuracy."},{"metadata":{},"cell_type":"markdown","source":"## Convolutional neural network \n\nConvolutional neural networks are a type of neural netork generally used for image recognition problems. But, the 1D version of CNNs can also be used for text-related problems (natural language processing). Convolution involves a process called convolution.\n\n### Convolution\n\nConvolution is a rather simple algorithm which involves a kernel (a 2D matrix) which moves over the entire matrix (word embeddings), calculating dot products with each window along the way. The GIF below demonstrates convolution in action.\n\n![](\"https://i.imgur.com/wYUaqR3.gif\")
\n\nThe above process can be summarized with an equation, where *f* is the image and *h* is the kernel. The dimensions of *f* are *(m, n)* and the kernel is a square matrix with dimensions smaller than *f*:\n\n![](\"https://i.imgur.com/9scTOGv.png\")
\n
\n\nIn the above equation, the kernel *h* is moving across the length and breadth of the matrix. The dot product of *h* with a sub-matrix or window of matrix *f* is taken at each step, hence the double summation (rows and columns).\n\n**In text classification, a 1D variant of convolution is used where the kernel moves in only one dimension.**"},{"metadata":{},"cell_type":"markdown","source":"I will onace again use the pretrained BERT embeddings as input, pass the embeddings through convolutional layers, and get the probability of the comment being toxic. The approach can be summarized using the flowchart below:\n\n![](\"https://i.imgur.com/7hsdV9T.png\")
"},{"metadata":{},"cell_type":"markdown","source":"The same logic is implemented below using tf.keras."},{"metadata":{},"cell_type":"markdown","source":"### Define CNN model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_cnn_model(transformer, max_len):\n input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n \n embed = transformer.weights[0].numpy()\n embedding = Embedding(np.shape(embed)[0], np.shape(embed)[1],\n input_length=max_len, weights=[embed],\n trainable=False)(input_word_ids)\n \n embedding = SpatialDropout1D(0.3)(embedding)\n conv_1 = Conv1D(64, 2)(embedding)\n conv_2 = Conv1D(64, 3)(embedding)\n conv_3 = Conv1D(64, 4)(embedding)\n conv_4 = Conv1D(64, 5)(embedding)\n \n maxpool_1 = GlobalAveragePooling1D()(conv_1)\n maxpool_2 = GlobalAveragePooling1D()(conv_2)\n maxpool_3 = GlobalAveragePooling1D()(conv_3)\n maxpool_4 = GlobalAveragePooling1D()(conv_4)\n conc = concatenate([maxpool_1, maxpool_2, maxpool_3, maxpool_4], axis=1)\n\n conc = Dense(64, activation='relu')(conc)\n conc = Dense(1, activation='sigmoid')(conc)\n \n model = Model(inputs=input_word_ids, outputs=conc)\n \n model.compile(Adam(lr=0.01), \n loss='binary_crossentropy', \n metrics=['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build model and check summary"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n model_cnn = build_cnn_model(transformer_layer, max_len=512)\n\nmodel_cnn.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize network architecture"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model_cnn, dpi=70).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_history = model_cnn.fit(\n train_dataset,\n steps_per_epoch=N_STEPS,\n validation_data=valid_dataset,\n callbacks = calls,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model predictions\n\nNow, I will visualize the performance of the model on few validation samples."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"visualize_model_preds(model_cnn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the Conv1D model gives much more accurate predictions than the the VNN model. It predicts toxicity correctly for all four validation samples, because it is able to understand spatial relationships better."},{"metadata":{},"cell_type":"markdown","source":"## LSTM with Attention \n \n\n### LSTM\n\nLSTMs are a type of neural network specifically made for NLP (text-related) tasks. In fact, LSTMs are a specific type of RNN. An RNN is a type of neural network that has a sense of direction (sequence). Classic neural networks look at all inputs at the same level, but RNNs look at inputs in a sequential order, which works well for text, as it is a sequential form of input.\n\nBut, RNNs have a problem called \"vanishing gradients\", which makes it difficult for it to understand long-term dependencies in text. Below is a depiction of the LSTM architecture which solves the problem of long-term dependencies:\n\n![](\"https://i.imgur.com/gmijcvr.png\")
\n\n### Attention\n\nAttention is a mathematical mechanism that allows a neural network to select its main areas of focus ina sequence. Understanding which part of the comment to focus on (based on mathematics and probabilities) can be crucial in predicting whether it is toxic or not. The Attention mechanism can be combined with LSTMs to produce excellent NLP models.\n\nThe approach can be summarized with flowchart below:\n\n\n![](\"https://i.imgur.com/SbFlht3.png\")
"},{"metadata":{},"cell_type":"markdown","source":"### Define the Attention layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"class AttentionWeightedAverage(Layer):\n\n def __init__(self, return_attention=False, **kwargs):\n self.init = initializers.get('uniform')\n self.supports_masking = True\n self.return_attention = return_attention\n super(AttentionWeightedAverage, self).__init__(** kwargs)\n\n def build(self, input_shape):\n self.input_spec = [InputSpec(ndim=3)]\n assert len(input_shape) == 3\n\n self.W = self.add_weight(shape=(input_shape[2], 1),\n name='{}_W'.format(self.name),\n initializer=self.init)\n super(AttentionWeightedAverage, self).build(input_shape)\n\n def call(self, x, mask=None):\n logits = K.dot(x, self.W)\n x_shape = K.shape(x)\n logits = K.reshape(logits, (x_shape[0], x_shape[1]))\n ai = K.exp(logits - K.max(logits, axis=-1, keepdims=True))\n\n if mask is not None:\n mask = K.cast(mask, K.floatx())\n ai = ai * mask\n att_weights = ai / (K.sum(ai, axis=1, keepdims=True) + K.epsilon())\n weighted_input = x * K.expand_dims(att_weights)\n result = K.sum(weighted_input, axis=1)\n if self.return_attention:\n return [result, att_weights]\n return result\n\n def get_output_shape_for(self, input_shape):\n return self.compute_output_shape(input_shape)\n\n def compute_output_shape(self, input_shape):\n output_len = input_shape[2]\n if self.return_attention:\n return [(input_shape[0], output_len), (input_shape[0], input_shape[1])]\n return (input_shape[0], output_len)\n\n def compute_mask(self, input, input_mask=None):\n if isinstance(input_mask, list):\n return [None] * len(input_mask)\n else:\n return None","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define the LSTM model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_lstm_model(transformer, max_len):\n input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n \n embed = transformer.weights[0].numpy()\n embedding = Embedding(np.shape(embed)[0], np.shape(embed)[1],\n input_length=max_len, weights=[embed],\n trainable=False)(input_word_ids)\n \n embedding = SpatialDropout1D(0.3)(embedding)\n lstm_1 = LSTM(128, return_sequences=True)(embedding)\n lstm_2 = LSTM(128, return_sequences=True)(lstm_1)\n \n attention = AttentionWeightedAverage()(lstm_2)\n conc = Dense(64, activation='relu')(attention)\n conc = Dense(1, activation='sigmoid')(conc)\n \n model = Model(inputs=input_word_ids, outputs=conc)\n \n model.compile(Adam(lr=0.01), \n loss='binary_crossentropy', \n metrics=['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build the model and check summary"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n model_lstm = build_lstm_model(transformer_layer, max_len=512)\n\nmodel_lstm.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize network architecture"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model_lstm, dpi=70).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_history = model_lstm.fit(\n train_dataset,\n steps_per_epoch=N_STEPS,\n validation_data=valid_dataset,\n callbacks = calls,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model predictions\n\n**Note: There are some errors which I am trying to fix here. The predictions cannot be visualized correctly.** But, I expect that LSTMs would perform well as they can understand text data well with attention."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"visualize_model_preds(model_lstm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Capsule network \n\nCNNs can extract features, but they cannot understand the spatial and proportional relationaships between objects in the image. Capsule networks solves this problem by understanding the spatial relationships between words (in text) by encoding additional information. A lot of the intuition behind how and why Capsule networks work is linked to image recognition, but it can also be used for text-based problems. \n\n![](\"https://i.imgur.com/oWLRwwA.png\")
"},{"metadata":{},"cell_type":"markdown","source":"For example, a CNN may be fooled into thinking the above image is a face as it cannot understand spatial relationships. But a Capsule network can understand these relationships and understand that this is not a face. The same logic applies for text as well. The approach can be summarized with the flowchart below:\n\n![](\"https://i.imgur.com/FkyE6bA.png\")
"},{"metadata":{},"cell_type":"markdown","source":"### Define the capsule layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"def squash(x, axis=-1):\n s_squared_norm = K.sum(K.square(x), axis, keepdims=True) + K.epsilon()\n scale = K.sqrt(s_squared_norm) / (0.5 + s_squared_norm)\n return scale * x\n\nclass Capsule(Layer):\n\n def __init__(self,\n num_capsule,\n dim_capsule,\n routings=3,\n share_weights=True,\n initializer='glorot_uniform',\n activation=None,\n regularizer=None,\n constraint=None,\n **kwargs):\n super(Capsule, self).__init__(**kwargs)\n self.num_capsule = num_capsule\n self.dim_capsule = dim_capsule\n self.routings = routings\n self.share_weights = share_weights\n\n self.activation = activations.get(activation)\n self.regularizer = regularizers.get(regularizer)\n self.initializer = initializers.get(initializer)\n self.constraint = constraints.get(constraint)\n\n def build(self, input_shape):\n input_dim_capsule = input_shape[-1]\n if self.share_weights:\n self.W = self.add_weight(name='capsule_kernel',\n shape=(1,\n input_dim_capsule,\n self.num_capsule *\n self.dim_capsule),\n initializer=self.initializer,\n regularizer=self.regularizer,\n constraint=self.constraint,\n trainable=True)\n else:\n input_num_capsule = input_shape[-2]\n self.W = self.add_weight(name='capsule_kernel',\n shape=(input_num_capsule,\n input_dim_capsule,\n self.num_capsule *\n self.dim_capsule),\n initializer=self.initializer,\n regularizer=self.regularizer,\n constraint=self.constraint,\n trainable=True)\n\n self.build = True\n\n def call(self, inputs):\n if self.share_weights:\n u_hat_vectors = K.conv1d(inputs, self.W)\n else:\n u_hat_vectors = K.local_conv1d(inputs, self.W, [1], [1])\n\n batch_size = K.shape(inputs)[0]\n input_num_capsule = K.shape(inputs)[1]\n u_hat_vectors = K.reshape(u_hat_vectors, (batch_size,\n input_num_capsule,\n self.num_capsule,\n self.dim_capsule))\n\n u_hat_vectors = K.permute_dimensions(u_hat_vectors, (0, 2, 1, 3))\n routing_weights = K.zeros_like(u_hat_vectors[:, :, :, 0])\n\n for i in range(self.routings):\n capsule_weights = K.softmax(routing_weights, 1)\n outputs = K.batch_dot(capsule_weights, u_hat_vectors, [2, 2])\n if K.ndim(outputs) == 4:\n outputs = K.sum(outputs, axis=1)\n if i < self.routings - 1:\n outputs = K.l2_normalize(outputs, -1)\n routing_weights = K.batch_dot(outputs, u_hat_vectors, [2, 3])\n if K.ndim(routing_weights) == 4:\n routing_weights = K.sum(routing_weights, axis=1)\n\n return self.activation(outputs)\n\n def compute_output_shape(self, input_shape):\n return (None, self.num_capsule, self.dim_capsule)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_capsule_model(transformer, max_len):\n input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n \n embed = transformer.weights[0].numpy()\n embedding = Embedding(np.shape(embed)[0], np.shape(embed)[1],\n input_length=max_len, weights=[embed],\n trainable=False)(input_word_ids)\n \n embedding = SpatialDropout1D(0.3)(embedding)\n capsule = Capsule(num_capsule=5, dim_capsule=5,\n routings=4, activation=squash)(embedding)\n\n capsule = Flatten()(capsule)\n output = Dense(128, activation='relu')(capsule)\n output = Dense(1, activation='sigmoid')(output)\n \n model = Model(inputs=input_word_ids, outputs=output)\n \n model.compile(Adam(lr=1.5e-5), \n loss='binary_crossentropy', \n metrics=['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build the model and check summary"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n model_capsule = build_capsule_model(transformer_layer, max_len=512)\n\nmodel_capsule.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_history = model_capsule.fit(\n train_dataset,\n steps_per_epoch=N_STEPS,\n validation_data=valid_dataset,\n callbacks = calls,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model architecture"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model_capsule, dpi=70).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model predictions"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"visualize_model_preds(model_capsule)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the Capsule layer gets only 2 out of 4 predictions correctly. Maybe, this architecture is not well-suited to this text-related problem."},{"metadata":{},"cell_type":"markdown","source":"## DistilBERT \n\n### BERT\n\nBERT (Bidirectional Encoder Representations from Transformers) was a paper published by researchers at Google AI Language, which caused a great stir in the NLP community as it became the SOTA on several NLP tasks.\n\nBERT’s key technical innovation is applying the bidirectional training of Transformer, a popular attention model, to language modelling. This is in contrast to previous efforts which looked at a text sequence either from left to right or combined left-to-right and right-to-left training (such as LSTMs). \n\nThe paper’s results show that a language model which is bidirectionally trained can have a deeper sense of language context and flow than single-direction language models. In the paper, the researchers detail a novel technique named Masked LM (MLM) which allows bidirectional training in models in which it was previously impossible.\n\n### DistilBERT\n\nDistilBERT is a lighter version of BERT (a very complex model) which uses fewer weights and achieves similar accuracies on several tasks with much lower training times. For this notebook, I will be using DistilBERT as it is easier to train in less time. The approach can be summarized with the flowchart below:\n\n![](\"https://i.imgur.com/6AGu9a4.png\")
"},{"metadata":{},"cell_type":"markdown","source":"### Define the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_distilbert_model(transformer, max_len=512):\n input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name=\"input_word_ids\")\n sequence_output = transformer(input_word_ids)[0]\n cls_token = sequence_output[:, 0, :]\n cls_token = Dense(500, activation=\"elu\")(cls_token)\n cls_token = Dropout(0.1)(cls_token)\n out = Dense(1, activation='sigmoid')(cls_token)\n \n model = Model(inputs=input_word_ids, outputs=out)\n \n model.compile(Adam(lr=1.5e-5), \n loss='binary_crossentropy', \n metrics=['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build the model and check summary"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n model_distilbert = build_distilbert_model(transformer_layer, max_len=512)\n\nmodel_distilbert.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_history = model_distilbert.fit(\n train_dataset,\n steps_per_epoch=N_STEPS,\n validation_data=valid_dataset,\n callbacks = calls,\n epochs=EPOCHS\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"SVG(tf.keras.utils.model_to_dot(model_distilbert, dpi=70).create(prog='dot', format='svg'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualize model predictions"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"visualize_model_preds(model_distilbert)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model correctly classifies only 3 out of 4 validation samples. Maybe the model needs more training to give better results as it is a very complex model."},{"metadata":{},"cell_type":"markdown","source":"### Generate submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(DATA_PATH + 'sample_submission.csv')\nsub['toxic'] = model_distilbert.predict(test_dataset, verbose=1)\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Takeaways \n\n1. Sentiment scores such as negativity and neutrality might be great features for classifying toxic comments.\n2. Several deep learning models can be used to classify toxic comments, such as Conv1D and LSTM.\n3. Pretrained models like BERT often give better results as they have excellent representational capacity. Pretrained models can easily adapt to unseen tasks with minimal data."},{"metadata":{},"cell_type":"markdown","source":"# Ending note \n\nPlease upvote this kernel if you like it. It motivates me to produce more quality content :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}