{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom scipy.stats.stats import pearsonr\n%matplotlib inline\nsns.set()\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.\nPATH = '/kaggle/input/jigsaw-multilingual-toxic-comment-classification/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to get a summary table for numeric columns and another one for object columns\ndef eda(df): \n    eda = df.describe().T\n    eda['null_sum'] = df.isnull().sum()\n    eda['null_pct'] = df.isnull().mean()\n    eda['dtypes'] = df.dtypes\n    \n    objects = df[[ x for x in df.columns if not x in eda.index]]\n    eda_objects = objects.describe().T\n    eda_objects['null_sum'] = df.isnull().sum()\n    eda_objects['null_pct'] = df.isnull().mean()\n    eda_objects['dtypes'] = df.dtypes\n    return eda, eda_objects","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA\nThis is rhe third of a series of NLP competition, hence the main data to consider here are obviously the comments. However, the datasets are enriched with some other variables that are worth exploring, just for the sake of learning. In this notebook I will just focus on those variables. Let's proceed."},{"metadata":{},"cell_type":"markdown","source":"## File 1 - Unintended Bias\nThis is an expanded version of the Civil Comments dataset with a range of additional labels.  \nSome basic data exploration."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train1 = pd.read_csv(PATH+'jigsaw-unintended-bias-train.csv')\ntrain1.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train1_eda, train1_eda_objects = eda(train1)\ntrain1_eda","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train1_eda_objects","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Knowing that target variable is 'toxic', there are several tipologies of columns that can be classified as:\n- ***Toxic Ratios***: Columns with values ranging [0,1] that are available for every row (no nulls). They clearly represent offensive comments. Columns are: severe_toxicity, obscene, identity_attack, insult, threat and sexual_explicit\n- ***Feature Ratios***: Columns with values ranging [0,1] that are not 'toxic ratios'. It appears that those ratios are available only for less than 25% of the rows though.\n- ***ID's***: Basically publication_id, parent_id and article_id. We'll get to them later\n- ***User reactions***: funny, wow, sad, likes and disagree. These seem to be the reaction to a comment. No nulls in these columns\n- ***Others***: These are comment_text (main column with texts, we are not going to analyze it here), created_date and rating\n"},{"metadata":{},"cell_type":"markdown","source":"Target variable in this training dataset is not 0 nor 1 but the probability bewtween 0 and 1. Let's check out the distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(10,6), nrows=1, ncols=2)\nfig.suptitle(\"Distribution of Target Variable\", size=25)\nsns.distplot(train1['toxic'], kde=False, bins=20, ax=ax[0])\nax[0].set(xlabel='Distribution')\nsns.distplot(train1['toxic'], kde=False, bins=[0,0.5,1], ax=ax[1])\nax[1].set(xlabel='Treshold = 0.5')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The target variable distribution is not normal, being most of the cases biased towards 0, meaning that the most common value is 0 or no toxic. The second figure shows the distribution with an hypothetical threshold of 0.5"},{"metadata":{},"cell_type":"markdown","source":"## Toxic Ratios\nColumns with values ranging [0,1] that are available for every row (no nulls). They clearly represent offensive comments. Columns are: severe_toxicity, obscene, identity_attack, insult, threat and sexual_explicit. Let's examine how they correlate to each other."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Features \ntoxic_ratios = ['severe_toxicity', 'obscene', 'identity_attack', 'insult', 'threat', 'sexual_explicit']\nfig = plt.figure(figsize=(8,8))\ntrain1_ratios = train1[toxic_ratios]\nsns.pairplot(train1_ratios)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The variables show low correlation. Maybe 'severe_toxicity' shows that it does not go beyond 0.5 too often. Let's check how they correlate against the target variable."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15,10), nrows=2,ncols=3)\nfor i,t in enumerate(toxic_ratios):\n    r,c = int(i/3),int(i%3)\n    sns.scatterplot(x=t, y=\"toxic\", data=train1, ax=ax[r][c])\n    ax[r][c].set(xlabel=t)\n    ax[r][c].plot([0,1], color='red')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems there is a pattern in all these variables. The value of the target variable is in most of cases (with some exceptions) greater than or equal to the value of the variable. Exceptions are those points below the red line in each figure. Let's find out how many exceptions we have in each case"},{"metadata":{"trusted":true},"cell_type":"code","source":"exceptions = []\nfor i,t in enumerate(toxic_ratios):\n    c = len(train1[train1[t]>train1['toxic']])\n    exceptions.append({'feature':t,'count': c, 'pct': c/len(train1)})\npd.DataFrame(exceptions).set_index('feature')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In most variables, less than 1% of cases happens to have a higher value than its corresponding toxic value. Can we consider them as outliers?"},{"metadata":{},"cell_type":"markdown","source":"Finally, adding them out in a single variable and plotting vs target feature as we just did above, yields the following result:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train1['ratios'] = train1[toxic_ratios].sum(axis=1)\nfig = plt.figure(figsize=(8,8))\nsns.scatterplot(x='ratios', y=\"toxic\", data=train1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature Ratios\nColumns with values ranging [0,1] that are not 'toxic ratios'. It appears that those ratios are available only for less than 25% of the rows though."},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_ratios = list(train1_eda[train1_eda['null_sum']>1000000].index) \ntrain1_ratios = train1[feature_ratios + ['toxic']].dropna()\ntrain1_ratios.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(21,14), nrows=4,ncols=6)\nfor i,t in enumerate(feature_ratios):\n    r,c = int(i/6),int(i%6)\n    sns.scatterplot(x=t, y=\"toxic\", data=train1_ratios, ax=ax[r][c])\n    ax[r][c].set(xlabel=t)\n    ax[r][c].plot([0,1], color='red')\n    plt.subplots_adjust(hspace=0.5, wspace= 0.5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the figures above we can't see the same pattern as with the \"toxic_ratios\"."},{"metadata":{"trusted":true},"cell_type":"code","source":"correlations = train1_ratios.corrwith(train1_ratios['toxic']).iloc[:-1].to_frame()\nsorted_correlations = correlations[0].sort_values(ascending=False)\nfig, ax = plt.subplots(figsize=(5,10))\nsns.heatmap(sorted_correlations.to_frame(), cmap='coolwarm', annot=True, vmin=-1, vmax=1, ax=ax)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Correlations very low with the target variable. Probably this is the case becase data in these variables are very sparse. Values are defaulted to 0 unless there is some component of that feature in the comments. So let's check it out."},{"metadata":{"trusted":true},"cell_type":"code","source":"correlations = []\nfor i,t in enumerate(feature_ratios):\n    r,c = int(i/6),int(i%6)\n    corr = {'feature':t}\n    corr['original'] = pearsonr(train1_ratios['toxic'], train1_ratios[t])[0]\n    df = train1_ratios[train1_ratios[t]>0]\n    corr['filtered'] = pearsonr(df['toxic'], df[t])[0]\n    correlations.append(corr)\n    \ncorrelations = pd.DataFrame(correlations).set_index('feature')\ncorrelations['original'] = correlations['original']\ncorrelations['filtered'] = correlations['filtered']\ncorrelations","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(8,10))\nfig.suptitle('Pearson Correlation vs target BEFORE vs AFTER filtering zeros', size=25)\nfor t in correlations.index:\n    plt.plot([correlations.loc[t,'original'],correlations.loc[t,'filtered']], label=t)\nplt.legend(loc='center left', bbox_to_anchor=(1, 0.5))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Removing 0 values from the features have mixed effects. In some cases, it increases the positive correlation with the target, which means the existance of certain language components increases the chances of a comment to be toxic. However, correlations for most of features remain in a range very close to 0, which means little to no correlation at all."},{"metadata":{},"cell_type":"markdown","source":"## ID's\nBasically publication_id, parent_id and article_id.  \nApparently there is little to explore in ID columns. However, I would like to analyze the possible relationship between rows with same **parent_id**"},{"metadata":{"trusted":true},"cell_type":"code","source":"ids = ['id','publication_id', 'parent_id', 'article_id']\ntrain1[ids].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Clearly *id* is just the row id, so nothing to be considered. However, the other variables have a very different number of values. Feature ***publication_id*** is likely to include some information as *just* 53 different values exists."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(50,10)) # adjust the fig size to see everything\nsns.barplot(x=train1['publication_id'].value_counts().index, y=train1['publication_id'].value_counts())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the comments belong to roughly 10 publications."},{"metadata":{},"cell_type":"markdown","source":"## User reactions\nfunny, wow, sad, likes and disagree. These seem to be the reaction to a comment. No nulls in these columns. Each comment can have more than one reaction, so they are not mutually exclusive.\n\nFirst thing is to find out how many comments don't have any reaction at all."},{"metadata":{"trusted":true},"cell_type":"code","source":"reactions = ['funny', 'wow', 'sad', 'likes' , 'disagree']\ntrain1_reaction = train1[reactions]\ntrain1_reaction['nreactions'] = train1_reaction.sum(axis=1)\nn = len(train1_reaction[train1_reaction['nreactions']==0])\nprint('Number of comments without reaction: {}'.format(n))\nprint('Pctg of comments without reaction: {s:.3f}'.format(s=n/len(train1)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Almost a third of the comments have no reaction at all. Let's check the correlations"},{"metadata":{"trusted":true},"cell_type":"code","source":"train1_reaction = train1[reactions]\ntrain1_reaction['nreactions'] = train1_reaction.sum(axis=1)\nsns.pairplot(train1_reaction[train1_reaction['nreactions']!=0].drop('nreactions',axis=1))\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Relationship between user reaction features is mostly inverse amongst them. This effect is more pronounced when reactions have a different sentiment, like funny vs disagree. When the sentiment is the same, like disagree vs sad, the effect is less pronounced (as expected) but still inverse. This suggests there is some sort of consensus among the users when rating comments.\n\nHowever, any single comment might have any variable number of reactions of each type, so establish an isolated relationship between each feature and the target variable could be tricky and misleading. Instead, I will assign a syntethic variable to each row based on the most repeated reaction (the feature with higher amount) and then check the correlation with the target variable."},{"metadata":{"trusted":true},"cell_type":"code","source":"train1_reaction = train1[reactions+['toxic']]\ntrain1_reaction['nreactions'] = train1_reaction.drop('toxic',axis=1).sum(axis=1)\ntrain1_reaction = train1_reaction[train1_reaction['nreactions']>0].drop('nreactions', axis=1)\ntrain1_reaction['reaction'] = train1_reaction.drop('toxic',axis=1).apply(lambda x: x.argmax(), axis=1 )\ntrain1_reaction['toxic_tr'] = train1_reaction['toxic'].apply(lambda x: int(1) if x>=0.5 else int(0) )\ngrouped = train1_reaction[['reaction','toxic_tr', 'wow']].groupby(['reaction', 'toxic_tr']).count().reset_index(drop=False).pivot(index='reaction', columns='toxic_tr', values='wow')\ngrouped['sum'] = grouped.sum(axis=1)\ngrouped[0] = 100*grouped[0]/grouped['sum']\ngrouped[1] = 100*grouped[1]/grouped['sum']\ngrouped = grouped.drop('sum', axis=1)\ngrouped","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure()\ncm = plt.get_cmap('viridis')\nax = fig.add_axes([0,0,1,1])\nax.set_title('$P( toxic | reaction=x)$', size=23)\ncolors = [ cm(i/(len(grouped.index))) for i in range(len(grouped.index))]\nlabels = grouped.index\nvalues = grouped[1]\nrects = ax.bar(labels, values, color=colors)\nfor p in rects:\n    ax.text( p.get_x() + p.get_width() / 2., p.get_height()* 1.05, s=str('{0:.2f}'.format(p.get_height())), ha = 'center', va = 'center')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above bars represent $P( toxic | reaction=x)$, in other words, the percentage of toxic comments (given our test threshold 0.5) when the main reacion is $x$.  \n\nThe insight here is that \"negative\" reactions (disagree, sad) are more likely to happen when the comment is toxic. I can't make any assumption about the relative low figures (less than 10%), but I would say that reaction is not a key variable in the toxicity of a comment."},{"metadata":{},"cell_type":"markdown","source":"## Others\nFeatures that are not numeric types but objects. Let's start with 'rating'."},{"metadata":{"trusted":true},"cell_type":"code","source":"train1['rating'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feature 'rating'just have 2 different values, so we can convert it to a binary variable for modelling. First, let's find out correlation with target variable."},{"metadata":{"trusted":true},"cell_type":"code","source":"train1['rating_binary'] = train1['rating'].apply(lambda x: 1 if x == 'approved' else 0)\nsns.boxplot(train1['rating_binary'], train1['toxic']).set_title('Toxic comments by Rating')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"We have converted value \"approved\" to 1 and \"rejected\" to 0, and we are checking the distribution of the target variable given each rating value. Distributions are different, which suggests that this feature could be of significance. Let's find out the Pearson correlation."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Pearson correlation between rating and target variable {s:.2f}\".format(s=pearsonr(train1['rating_binary'], train1['toxic'])[0]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Correlation is negative, as expected (it would have been positive if we would assign value 1 to 'rejected').  \n\nFinally, let's check created_date, to find out if there is any kind of seasonality or trend related with the target variable."},{"metadata":{"trusted":true},"cell_type":"code","source":"train1['created_date_date'] = pd.to_datetime(train1['created_date']).dt.date\ngrouped = train1.groupby('created_date_date').count()[['id']]\nfig = plt.figure(figsize=(20,5))\nax = sns.lineplot(x=grouped.index, y= grouped.id)\nax.set_title('Number of comments by date', size=23)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First thing we see is that the number of comments has been increasing. This is of little value apparently, because the dataset has been selected by the competition organizers, and this does not represent the whole comments population."},{"metadata":{},"cell_type":"markdown","source":"I wuld like to check if for specific periods, the trend is to post more toxic comments. I would also see if there is any seasonality in it. So I would group 'toxic' values by date and average them, so I have a '*toxicity average*' by date."},{"metadata":{"trusted":true},"cell_type":"code","source":"train1['created_date_date'] = pd.to_datetime(train1['created_date']).dt.date\ngrouped = train1[['created_date_date','toxic']].groupby('created_date_date').mean()\nfig = plt.figure(figsize=(20,6))\nax = sns.lineplot(x=grouped.index, y= grouped.toxic)\nax.set_title('Average Toxic by Date', size=23)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The time series seems to be stationary. No trend apparently, except for the initial months that there is some variance in the data. Let's decompose it."},{"metadata":{"trusted":true},"cell_type":"code","source":"from statsmodels.tsa.seasonal import seasonal_decompose\nfrom pylab import rcParams\nrcParams['figure.figsize'] = 20, 8\nresult = seasonal_decompose(grouped, model='additive', freq=1)\nfig = result.plot()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This plot is probably unable to decompose properly the series, so let's run ADF and KPSS tests on it"},{"metadata":{"trusted":true},"cell_type":"code","source":"from statsmodels.tsa.stattools import adfuller\nfrom statsmodels.tsa.stattools import kpss\n\ndef adf_test(timeseries):\n    print ('Results of Dickey-Fuller Test:')\n    dftest = adfuller(timeseries, autolag='AIC')\n    dfoutput = pd.Series(dftest[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])\n    for key,value in dftest[4].items():\n       dfoutput['Critical Value (%s)'%key] = value\n    print (dfoutput)\n    \ndef kpss_test(timeseries):\n    print ('Results of KPSS Test:')\n    kpsstest = kpss(timeseries, regression='c', nlags=None)\n    kpss_output = pd.Series(kpsstest[0:3], index=['Test Statistic','p-value','Lags Used'])\n    for key,value in kpsstest[3].items():\n        kpss_output['Critical Value (%s)'%key] = value\n    print (kpss_output)\n    \nadf_test(grouped)\nprint('*'*20)\nkpss_test(grouped)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ADF indicates the serie is stationary and KPSS indicates the opposite, so the series might be difference stationary"},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped['toxic_diff'] = grouped['toxic'] - grouped['toxic'].shift(1)\ngrouped['toxic_diff'].dropna().plot(figsize=(12,8))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"adf_test(grouped[['toxic_diff']].dropna())\nprint('*'*20)\nkpss_test(grouped[['toxic_diff']].dropna())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This time, both tests yield the series to be stationary, so the relationship between 'creation_date' and 'toxic' yield little information"},{"metadata":{},"cell_type":"markdown","source":"## File 2 - Toxic Comment\nThe dataset is made up of English comments from Wikipedia’s talk page edits."},{"metadata":{"trusted":true},"cell_type":"code","source":"train2 = pd.read_csv(PATH+'jigsaw-toxic-comment-train.csv')\ntrain2.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this file we just have what we called the 'toxic ratios' with slight differences. Feature 'identity_hate' is not present in the other dataset. At the same time, some other features from thic category present on the other dataset are not present here."},{"metadata":{"trusted":true},"cell_type":"code","source":"train2_eda, train2_eda_objects = eda(train2)\ntrain2_eda","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just from this summary we can see that numeric variables are all binary, unlike the previous dataset. So just by looking at the mean column, we can see the proportion of 1's and 0's for each variable."},{"metadata":{"trusted":true},"cell_type":"code","source":"train2_eda_objects","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So taking a look at the relationship between the target variable and each of the features"},{"metadata":{"trusted":true},"cell_type":"code","source":"toxic_ratios = ['severe_toxic', 'obscene', 'identity_hate', 'insult', 'threat']\nfig, ax = plt.subplots(figsize=(25,3), nrows=1,ncols=5)\nfor i,t in enumerate(toxic_ratios):\n    df = train2[['toxic',t,'id']].groupby(['toxic',t]).count().reset_index()\n    df = df.pivot(index='toxic', columns=t, values='id')\n    sns.heatmap( data=df, ax=ax[i], annot=True, fmt='.0f')\n    ax[i].set(xlabel=t)\n    plt.subplots_adjust( wspace= 0.5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The pattern is clear here. The existance of any of these features indicates a high chance of the comment to be toxic."},{"metadata":{"trusted":true},"cell_type":"code","source":"correlations = []\nfor i,t in enumerate(toxic_ratios):\n    corr = {'feature':t}\n    corr['correlation'] = pearsonr(train2['toxic'], train2[t])[0]\n    df = train2[train2[t]>0]\n    correlations.append(corr)\n    \ncorrelations = pd.DataFrame(correlations).set_index('feature')\ncorrelations","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next steps will be to explore the comments column.  \nIf you liked this notebook, please upvote!!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}