{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = \"/kaggle/input/jigsaw-comment-classification-notebook-one/\"\nos.listdir(DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_PATH = DATA_PATH + \"notebook_one_train_data.csv\"\nVAL_PATH = DATA_PATH + \"notebook_one_val_data.csv\"\nTEST_PATH = DATA_PATH + \"notebook_one_test_data.csv\"\n\ntrain_data = pd.read_csv(TRAIN_PATH)\nval_data = pd.read_csv(VAL_PATH)\ntest_data = pd.read_csv(TEST_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Translation \n\nIn the [first notebook of this notebook series](https://www.kaggle.com/sleebapaul/jigsaw-comment-classification-notebook-one), I've done some deductions based on the EDA. \n\nMain findings are, \n\n1. The training data is hightly skewed. 90% of the data is non-toxic comments.\n2. The non-english comments are only 1% of training data\n3. In contrast, in testing and validation data, all the comments are non-english. \n4. A viable solution is to translate these sentences to English and train the model. \n5. On testing, first translate the sentence to English and do the prediction. \n6. In support of this decision, most of the NLP packages available are trained and tuned in English language data.\n7. Thus processing the non-english sentences with these packages and arriving on a conclusion is meanless. \n\n\nFor detailed information, check out the [first notebook](https://www.kaggle.com/sleebapaul/jigsaw-comment-classification-notebook-one) mentioned above. \n\n\n\n# Translated Validation and Test Data\n\nTranslation of the validation and test data are already by [Yury Kashnitsky](https://www.kaggle.com/kashnitsky). The data is reused for further analysis. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_PATH = \"/kaggle/input/jigsaw-multilingual-toxic-test-translated/\"\nos.listdir(DATA_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRANSLATED_VAL_PATH = DATA_PATH + \"jigsaw_miltilingual_valid_translated.csv\"\nTRANSLATED_TEST_PATH = DATA_PATH + \"jigsaw_miltilingual_test_translated.csv\"\n\ntrans_val_data = pd.read_csv(TRANSLATED_VAL_PATH)\ntrans_test_data = pd.read_csv(TRANSLATED_TEST_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trans_val_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trans_test_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data[\"translated_comment\"] = trans_val_data.translated\ntest_data[\"translated_comment\"] = trans_test_data.translated","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# About training data\n\nNow we have the validation and testing data translations, now let's dig deep into the non-english comments in training data. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_non_eng_sentences = train_data[(train_data.lang_code != 'en')]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Total number of comments which are Non-English: \",\n      train_non_eng_sentences.shape[0])\n\nprint(\"Total number of languages other than English: \",\n      len(train_non_eng_sentences.lang_code.unique()))\n\nprint(\"Average number of comments in Non-English Languages: \",\n      train_non_eng_sentences.lang_code.value_counts().mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(12, 10), )\nprob = train_non_eng_sentences.lang_code.value_counts(normalize = True)\nthreshold = .01\nmask = prob > threshold\ntail_prob = prob.loc[~mask].sum()\nprob = prob.loc[mask]\nprob['other'] = tail_prob\nax = sns.barplot(x=prob.index, y=prob.values)\nax.set_xticklabels(ax.get_xticklabels(), rotation=10, ha=\"right\")\nax.set_xlabel(\"Languages\")\nax.set_ylabel(\"Comment count\")\nax.set_title('Non English language comment count in training data (>1% of total)', fontsize=14)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings\n\n1. The graph plot the non-english languages which has a count greater than atleadt 1% of total count.\n2. From 142 non english languages, 23 languages are eligible for that count. \n3. That means, most of the languages present are feeble in number (`other ~ 37%`). \n4. `un` category consists of those comments which couldn't be detected by `polyglot` package. \n5. More than 22% of all non-english comments comes under `un`. This category shoud be looked closely. \n6. Can we remove the rest of the languages other than English?\n\n    - It can found out from the amount at which they are contributing to `toxic` comments.\n    - Because, it is known that the dataset is heavily skewed with 99% of `non-toxic` comments.\n    - So we can't lose the data, if these comments comes under `toxic` category. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_non_eng_toxic = train_data[(train_data.lang_code != 'en') & (train_data.label == 1)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Total toxic comments available in the training dataset: \", train_data.label.value_counts()[1])\nprint(\"Total toxic comments that are non english: \", train_non_eng_toxic.shape[0])\nprint(\"Percetage contribution: {:.2f} %\".format(100*(\n    train_non_eng_toxic.shape[0]/train_data.label.value_counts()[1])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings \n\n1. 2% of non english comments are `toxic`. \n2. Let's save this data and chuck the rest, since we've abundant data for `non-toxic` comments already. \n3. Translate these 446 comments to English and drop the rest.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"! pip install googletrans","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from googletrans import Translator\n\ntranslator = Translator()\n\ndef translate(sentence):\n    return translator.translate(sentence).text","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings\n\n1. This is a very interesting finding that the examples with undetected language (`un`) are not really foreign.\n2. Almost all of them are English language with punctuation issues, HTML, repeated text etc. \n3. The good news is, they are need not to be removed from the dataset. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"translated_train_sents = []\n\ntrain_non_eng_toxic_comment_list = train_non_eng_toxic.comment_text.to_list()\nprint(\"Total length: \", len(train_non_eng_toxic_comment_list))\n\ndef clean_text_for_translate(comment):\n    if type(comment) == str:\n        x = \"\".join(x for x in comment if x.isprintable())        \n        return x.replace(\"\\n\", \" \")\n    else:\n        return \"\"\n\ncount = 0 \n\nwhile count < len(train_non_eng_toxic_comment_list):\n    sent = clean_text_for_translate(train_non_eng_toxic_comment_list[count])\n    translated_train_sents.append(translate(sent))\n    count += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.loc[(train_data.lang_code != 'en') & (train_data.label == 1), 'comment_text']= translated_train_sents\ntrain_data.loc[(train_data.lang_code != 'en') & (train_data.label == 1), 'lang_code'] = 'en'\ntrain_data.loc[(train_data.lang_code != 'en') & (train_data.label == 1), 'lang_name'] = 'English'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.drop(train_data[(train_data['lang_code'] != 'en') & (train_data['label'] == 0)].index, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# Cleaning the comments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install beautifulsoup4\n!pip install contractions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from bs4 import BeautifulSoup\nimport re\nimport contractions\nimport unicodedata\n\n\ndef remove_html(text):\n    soup = BeautifulSoup(text, \"html.parser\")\n    html_free = soup.get_text()\n    return html_free\n\ndef remove_url(text):\n    return re.sub(r'http\\S', ' ', text)\n\ndef remove_digits_spec_chars(text):\n    return re.sub(r'[^a-zA-Z]', \" \", text)\n\ndef to_lower_case(text):\n    return text.lower()\n\ndef remove_extra_spaces(text):\n    return re.sub(r'\\s\\s+', \" \", text)\n\ndef remove_next_line(text):\n    return re.sub(r'[\\n\\r]+', \" \", text)\n    \ndef remove_non_ascii(comment):\n    \"\"\"\n    Remove non-ASCII characters from list of tokenized words\n    \"\"\"\n    ascii_string = unicodedata.normalize('NFKD', comment).encode('ascii', 'ignore').decode('utf-8', 'ignore')\n    return ascii_string\n    \ndef remove_between_square_brackets(comment):\n    result = re.sub('\\[[^]]*\\]', '', comment)\n    return result\n\ndef replace_contractions(comment):\n    \"\"\"\n    Replace contractions in string of text\n    \"\"\"\n    contraction_free = contractions.fix(comment)\n    return contraction_free\n\ndef clean_comment(comment):\n    comment = remove_non_ascii(comment)\n    comment = remove_next_line(comment)\n    comment = replace_contractions(comment)\n    comment = remove_url(comment)\n    comment = remove_html(comment)\n    comment = remove_between_square_brackets(comment)\n    comment = remove_digits_spec_chars(comment)\n    comment = remove_extra_spaces(comment)\n    comment = to_lower_case(comment)\n    return comment.strip()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data[\"cleaned_text\"] = train_data[\"comment_text\"].apply(lambda row: clean_comment(row))\nval_data[\"cleaned_text\"] = val_data[\"translated_comment\"].apply(lambda row: clean_comment(row))\ntest_data[\"cleaned_text\"] = test_data[\"translated_comment\"].apply(lambda row: clean_comment(row))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Add sentiment data ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install --upgrade vaderSentiment","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n\nanalyzer = SentimentIntensityAnalyzer()\n\ndef get_pos_polarity(comment):\n    return analyzer.polarity_scores(comment)\n\ndef get_comment_length(comment):\n    try:\n        return len(comment.split())\n    except:\n        return 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sentiment = train_data[\"cleaned_text\"].apply(lambda row: get_pos_polarity(row))\ntrain_data = pd.concat([train_data,sentiment.apply(pd.Series)],1)\ntrain_data.columns = ['id', 'comment_text', 'toxic', 'severe_toxic', 'obscene', 'threat',\n       'insult', 'identity_hate', 'lang_code', 'lang_name', 'country', 'label', 'cleaned_text',\n                      'neg_pol','neutral_pol', 'pos_pol', 'compound_pol']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sentiment = val_data[\"cleaned_text\"].apply(lambda row: get_pos_polarity(row))\nval_data = pd.concat([val_data,sentiment.apply(pd.Series)],1)\nval_data.columns = ['id', 'comment_text', 'lang', 'toxic', 'lang_name', 'country',\n                    'translated_comment', 'cleaned_text','neg_pol', \n                    'neutral_pol', 'pos_pol', 'compound_pol']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sentiment = test_data[\"cleaned_text\"].apply(lambda row: get_pos_polarity(row))\ntest_data = pd.concat([test_data,sentiment.apply(pd.Series)],1)\ntest_data.columns = ['id', 'content', 'lang', 'lang_name', 'country', 'translated_comment',\n                    'cleaned_text','neg_pol', 'neutral_pol', 'pos_pol', 'compound_pol']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Add comment length to data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data[\"comment_len\"] = train_data[\"cleaned_text\"].apply(lambda row: get_comment_length(row))\nval_data[\"comment_len\"] = val_data[\"cleaned_text\"].apply(lambda row: get_comment_length(row))\ntest_data[\"comment_len\"] = test_data[\"cleaned_text\"].apply(lambda row: get_comment_length(row))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Comment length analysis","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, ax = plt.subplots(nrows=1, ncols=1, figsize=(8, 6), )\nax = sns.kdeplot(train_data.comment_len, shade=True, label = \"Training\")\nax = sns.kdeplot(val_data.comment_len, shade=True, label = \"Validation\")\nax = sns.kdeplot(test_data.comment_len, shade=True, label = \"Testing\")\nax.set_title('Density distribution of comment length over different datasets')\nax.set_xlabel(\"Comment length\")\nax.set_ylabel(\"Density\")\nf.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))\nsns.kdeplot(train_data[train_data.label ==1].comment_len, shade=True, label = \"Toxic Comment\", ax=ax1)\nsns.kdeplot(train_data[train_data.label ==0].comment_len, shade=True, label = \"Non-Toxic Comment\", ax=ax1)\nax1.set_title('Training data')\nax1.set_xlabel(\"Comment length\")\nax1.set_ylabel(\"Density\")\n\nsns.kdeplot(val_data[val_data.toxic ==1].comment_len, shade=True, label = \"Toxic Comment\", ax=ax2)\nsns.kdeplot(val_data[val_data.toxic ==0].comment_len, shade=True, label = \"Non-Toxic Comment\", ax=ax2)\nax2.set_title('Validation data')\nax2.set_xlabel(\"Comment length\")\nax2.set_ylabel(\"Density\")\n\n\nf.suptitle(\"Density distribution of comment length over labels\", fontsize = 22)\nf.tight_layout()\nf.subplots_adjust(top=0.8)\nf.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(18,6))\n\n# Training data \nvar = 'comment_len'\ntmp = pd.concat([train_data['label'], train_data[var]], axis=1)\nsns.boxplot(x='label', y=var, data=tmp, fliersize=5, ax=ax1)\nax1.set_title('Training data')\nax1.set_xlabel(\"Labels\")\nax1.set_ylabel(\"Comment Length\")\nax1.set_xticklabels([\"Non-Toxic\", \"Toxic\"])\n\ntmp = pd.concat([val_data['toxic'], val_data[var]], axis=1)\nsns.boxplot(x='toxic', y=var, data=tmp, fliersize=5, ax=ax2)\nax2.set_title('Validation data')\nax2.set_xlabel(\"Labels\")\nax2.set_ylabel(\"Comment Length\")\nax2.set_xticklabels([\"Non-Toxic\", \"Toxic\"])\n\nfig = sns.boxplot(y=var, data=test_data, fliersize=5, ax=ax3)\nax3.set_title('Test data')\nax3.set_ylabel(\"Comment Length\")\n\nf.suptitle(\"Comment count across datasets\", fontsize = 22)\nf.tight_layout()\nf.subplots_adjust(top=0.8)\nf.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Q1 = train_data.comment_len.quantile(0.25)\nQ3 = train_data.comment_len.quantile(0.75)\nIQR = Q3 - Q1\n\nprint(\"Total comment length outliers in training data: \", \n      ((train_data.comment_len < (Q1 - 1.5 * IQR)) | (train_data.comment_len > (Q3 + 1.5 * IQR))).sum())\n\nprint(\"Total comment length outliers in toxic comments of training data: \", \n      ((train_data.loc[train_data.label == 1]['comment_len'] < (Q1 - 1.5 * IQR))|(\n          train_data.loc[train_data.label == 1]['comment_len'] > (Q3 + 1.5 * IQR))).sum())\n\nprint(\"Total comment length outliers in non-toxic comments of training data: \", \n      ((train_data.loc[train_data.label == 0]['comment_len'] < (Q1 - 1.5 * IQR))|(\n          train_data.loc[train_data.label == 0]['comment_len'] > (Q3 + 1.5 * IQR))).sum())\n\n\nprint(\"-\"*80)\n\nQ1 = val_data.comment_len.quantile(0.25)\nQ3 = val_data.comment_len.quantile(0.75)\nIQR = Q3 - Q1\n\nprint(\"Total comment length outliers in validation data: \", \n      ((val_data.comment_len < (Q1 - 1.5 * IQR)) | (val_data.comment_len > (Q3 + 1.5 * IQR))).sum())\n\nprint(\"Total comment length outliers in toxic comments of validation data: \", \n      ((val_data.loc[val_data.toxic == 1]['comment_len'] < (Q1 - 1.5 * IQR))|(\n          val_data.loc[val_data.toxic == 1]['comment_len'] > (Q3 + 1.5 * IQR))).sum())\n\nprint(\"Total comment length outliers in non-toxic comments of validation data: \", \n      ((val_data.loc[val_data.toxic == 0]['comment_len'] < (Q1 - 1.5 * IQR))|(\n          val_data.loc[val_data.toxic == 0]['comment_len'] > (Q3 + 1.5 * IQR))).sum())\n\n\nprint(\"-\"*80)\n\nQ1 = test_data.comment_len.quantile(0.25)\nQ3 = test_data.comment_len.quantile(0.75)\nIQR = Q3 - Q1\n\nprint(\"Total comment length outliers in testing data: \", \n      ((test_data.comment_len < (Q1 - 1.5 * IQR)) | (test_data.comment_len > (Q3 + 1.5 * IQR))).sum())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings\n\n1. Density distribution of comment length over datasets are almost similar. No significance change is observed. \n2. Having said that, training data is more right skewed than any other data. \n3. Density distributions of comment lengths over toxic/non-toxic comments, are also similar. Comment length is not significatly seperating the two labels. \n4. Coming to outliers,\n    - Validation set comment length ranges upto 350 and in test data it is 800.\n    - But in training data, this range is upto 2000+. \n    - There are comments in training data which are significantly larger comparing to other datasets.\n    - Removing outliers might reduce the number of toxic comments as well, thus keeping them in the dataset. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Sentiments \n\n1. There are four components recorded for a sentence in the dataset\n2. Positive, Negative, Neutral and compound. \n\n\n## Negative sentiment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))\nsns.kdeplot(train_data[train_data.label ==1].neg_pol, shade=True, label = \"Toxic Comment\", ax=ax1)\nsns.kdeplot(train_data[train_data.label ==0].neg_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax1)\nax1.set_title('Training data')\nax1.set_xlabel(\"Negative Polarity\")\nax1.set_ylabel(\"Density\")\n\nsns.kdeplot(val_data[val_data.toxic ==1].neg_pol, shade=True, label = \"Toxic Comment\", ax=ax2)\nsns.kdeplot(val_data[val_data.toxic ==0].neg_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax2)\nax2.set_title('Validation data')\nax2.set_xlabel(\"Negative Polarity\")\nax2.set_ylabel(\"Density\")\n\n\nf.suptitle(\"Density distribution of Negative Polarity over labels\", fontsize = 22)\nf.tight_layout()\nf.subplots_adjust(top=0.8)\nf.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Positive sentiments","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))\nsns.kdeplot(train_data[train_data.label ==1].pos_pol, shade=True, label = \"Toxic Comment\", ax=ax1)\nsns.kdeplot(train_data[train_data.label ==0].pos_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax1)\nax1.set_title('Training data')\nax1.set_xlabel(\"Positive Polarity\")\nax1.set_ylabel(\"Density\")\n\nsns.kdeplot(val_data[val_data.toxic ==1].pos_pol, shade=True, label = \"Toxic Comment\", ax=ax2)\nsns.kdeplot(val_data[val_data.toxic ==0].pos_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax2)\nax2.set_title('Validation data')\nax2.set_xlabel(\"Positive Polarity\")\nax2.set_ylabel(\"Density\")\n\n\nf.suptitle(\"Density distribution of Positive Polarity over labels\", fontsize = 22)\nf.tight_layout()\nf.subplots_adjust(top=0.8)\nf.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Neutral polarity","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))\nsns.kdeplot(train_data[train_data.label ==1].neutral_pol, shade=True, label = \"Toxic Comment\", ax=ax1)\nsns.kdeplot(train_data[train_data.label ==0].neutral_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax1)\nax1.set_title('Training data')\nax1.set_xlabel(\"Neutral Polarity\")\nax1.set_ylabel(\"Density\")\n\nsns.kdeplot(val_data[val_data.toxic ==1].neutral_pol, shade=True, label = \"Toxic Comment\", ax=ax2)\nsns.kdeplot(val_data[val_data.toxic ==0].neutral_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax2)\nax2.set_title('Validation data')\nax2.set_xlabel(\"Neutral Polarity\")\nax2.set_ylabel(\"Density\")\n\n\nf.suptitle(\"Density distribution of Neutral Polarity over labels\", fontsize = 22)\nf.tight_layout()\nf.subplots_adjust(top=0.8)\nf.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Compound polarity","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12,6))\nsns.kdeplot(train_data[train_data.label ==1].compound_pol, shade=True, label = \"Toxic Comment\", ax=ax1)\nsns.kdeplot(train_data[train_data.label ==0].compound_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax1)\nax1.set_title('Training data')\nax1.set_xlabel(\"Compound Polarity\")\nax1.set_ylabel(\"Density\")\n\nsns.kdeplot(val_data[val_data.toxic ==1].compound_pol, shade=True, label = \"Toxic Comment\", ax=ax2)\nsns.kdeplot(val_data[val_data.toxic ==0].compound_pol, shade=True, label = \"Non-Toxic Comment\", ax=ax2)\nax2.set_title('Validation data')\nax2.set_xlabel(\"Compound Polarity\")\nax2.set_ylabel(\"Density\")\n\n\nf.suptitle(\"Density distribution of Compound Polarity over labels\", fontsize = 22)\nf.tight_layout()\nf.subplots_adjust(top=0.8)\nf.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Findings\n\n1. As expected, negative polarity is significantly seperating the comments to toxic and non-toxic. \n2. For non-toxic comments, negative polarity is near to zero. \n3. Surprisingly, positive polarity is not adding much value to seperate the comments. The distribution is almost identical for both groups. \n4. This can be reasoned using the following plots.\n    - Both types of comments are giving small values (0.0 ~ 0.2) for positive polarities. \n    - Both types of comments are giving high neutral polarity sentiment score. Comparitively, non-toxic comments \n    get high neutral values. \n    - Thus, the sentiment analyser is classifying most of the comments in the data as neutral comments.\n    - This can be confirmed in next plot of compound polarity.\n    - Signficant amount of data aligns between `-0.5 to 0.5` which is by definition, neutral. \n    \n5. Negative and Compound sentiments are contributing features to classify the comments. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Saving the notebooks","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.to_csv(\"notebook_two_train_data.csv\", index = False)\nval_data.to_csv(\"notebook_two_val_data.csv\", index = False)\ntest_data.to_csv(\"notebook_two_test_data.csv\", index = False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}