{"cells":[{"metadata":{"_uuid":"0f9afca5816a9a32703f74778d803d40fc48ffee"},"cell_type":"markdown","source":"# Profanity in Quora questions\nThis notebook investigates the use and distribution of profanity (offensive language) among the Quora dataset of questions. Perhaps detection of profaniity in a question could help determine whether the question is insincere. To flag inappropriate language, we use the list of words blacklisted by Google, found [here](https://www.freewebheaders.com/full-list-of-bad-words-banned-by-google/). Throughout the notebook, these are referred to as 'bad words'.\n\nPlease leave advice / comments below and feel free to fork.\n\nWarning: This notebook displays language that may be offensive.\n"},{"metadata":{"_uuid":"a29a36a1b12e561686f0ba6018bbc7a86d202a1c"},"cell_type":"markdown","source":"## Import packages and data"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":false,"_kg_hide-output":false,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# Import packages\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport pickle\nfrom nltk.tokenize import word_tokenize\nfrom tqdm.auto import tqdm\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Import training and test data from Quora questions dataset\ntrain = pd.read_csv('../input/quora-insincere-questions-classification/train.csv')\ntest = pd.read_csv('../input/quora-insincere-questions-classification/test.csv')\n\n# Import list of bad words (banned by Google)\nbad_words = pd.read_csv('../input/bad-words/bad_words.csv', header=None)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7b682efa9fea7121295f0f4756436abd4bc1f3f","trusted":true},"cell_type":"code","source":"# Sample of bad words - some are actually not swear words as such\nbad_words.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af0ba19ad79e150d175cefb77c5b8878ab91f83b"},"cell_type":"markdown","source":"## Function to detect and store bad words from a given sentence"},{"metadata":{"_uuid":"bf8b7227636bed3a566a171d07c455bc7907287b","trusted":false},"cell_type":"code","source":"# Returns a list (possibly empty) of bad words in text\ndef detect_badwords(text):\n    # tokenize the text\n    tokens = word_tokenize(text)\n    bad_found = []\n    for word in tokens:\n        for bad_word in bad_words[0]:\n            if bad_word == word:\n                bad_found.append(word)\n    return bad_found","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"babfa0bc1a1337267e64feedc98872f01c1cf6c3","trusted":false},"cell_type":"code","source":"# Simple examples\ns1 = 'This politician speaks nothing but bullshit, what a twat'\ns2 = 'The sun shines brightly today'\n[detect_badwords(s) for s in [s1,s2]]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"851aedc159fc2b3c17a67544e45803b9cb52335b"},"cell_type":"markdown","source":"## Apply function to the Quora dataset\n"},{"metadata":{"_uuid":"2db70122fbd5126211f07ed0c6f1f83c23042446","trusted":false},"cell_type":"code","source":"# Register `pandas.progress_apply' with `tqdm`\ntqdm.pandas()\n# Run detect_badwords over all entries in the training data\ntemp_size = 10000\ntrain_bad_words = train['question_text'].progress_apply(detect_badwords)\n# Add as a column to the DataFrame\ntrain['bad_words'] = train_bad_words","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2aeea2c1d274afd74b81ad096c823bf013dff065","scrolled":true,"trusted":false},"cell_type":"code","source":"# Collect all entries that contain bad words\nbools_bw_nonempty = [train['bad_words'].iloc[i] != [] for i in tqdm(range(train.shape[0]))]\ndf_bw = train[bools_bw_nonempty]\n\n# Sincere set\ndf_bw_sincere = df_bw[df_bw['target'] == 0]\n# Insincere set\ndf_bw_insincere = df_bw[df_bw['target'] ==1 ]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"323764c2b9bd3405b64dd601b77a86225d64edde"},"cell_type":"markdown","source":"### Sample of sincere posts containing bad words"},{"metadata":{"_uuid":"2f451cc032ed10a2983cdea19cef09ba7c321cc3","scrolled":true,"trusted":false},"cell_type":"code","source":"df_bw_sincere.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3734f595ad2a35eeff061349d66c5d2747505d5f"},"cell_type":"markdown","source":"### Sample of insincere posts containing bad words"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"15d3b684e87c321114d372a1f02a4e80b5bb965c"},"cell_type":"code","source":"df_bw_insincere.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53c8a4412458a36b662a25f98d0b7ab30bdee72d"},"cell_type":"markdown","source":"## Count of posts with bad words"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"c1bbefdb05a1f9c8828b2b04e65f2f49e22d7d69"},"cell_type":"code","source":"# Total number of entries that use at least one bad word\nbw_total_sincere = len(df_bw_sincere)\nbw_total_insincere = len(df_bw_insincere)\n\nprint('Total number of sincere posts that contain at least one bad word:', bw_total_sincere)\nprint('Total number of insincere posts that contain at least one bad word:', bw_total_insincere)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b936dd4ea35612f1232ee0f60f61d186cec39c85"},"cell_type":"code","source":"# Total number of sincere and insincere posts\nnum_sincere = len(train[ train['target'] == 0])\nnum_insincere = len(train[ train['target'] == 1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"68a46886f73c6163aedfa9d4e7d20847b53e9fa9"},"cell_type":"code","source":"# Proportions\nbw_ratio_sincere = bw_total_sincere/num_sincere\nbw_ratio_insincere = bw_total_insincere/num_insincere\n\n# Plot\nratio_data = pd.DataFrame(\n    {'Flag': ['Sincere', 'Insincere'],\n     'Proportion': [bw_ratio_sincere, bw_ratio_insincere]}\n)\nratio_data","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"2b809a8ab5e76197007c2397403b164306e5488b"},"cell_type":"code","source":"# Plot of proportion of questions that contain bad words\nsns.barplot(x='Flag', \n            y='Proportion',\n            data=ratio_data).set_title('Proportion of entries with at least one bad word');","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"29c61ed2a7077cb9e153de042dfc124b0a020db1"},"cell_type":"code","source":"print('''From the training data, insincere posts are about\n{:.1f} times more likely to contain bad words'''.format(bw_ratio_insincere/bw_ratio_sincere))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f33cd39933a5382a37fb9141ef9de3f590a92de"},"cell_type":"markdown","source":"There are many sincere posts that also contain bad words. Perhaps most of these words in the sincere posts are not so bad. Maybe we can extract the particularly vulgar words."},{"metadata":{"_uuid":"7f819ec9e9a759073277cb087c6a04ee97165d9b"},"cell_type":"markdown","source":"## Distribution among the bad words"},{"metadata":{"trusted":false,"_uuid":"f9d9eb271ccd8aef62aac821a99e001a4c77be5f"},"cell_type":"code","source":"# Define a function to flatten a list of lists\ndef flatten(l):\n    x = []\n    for sublist in l:\n        for element in sublist:\n            x.append(element)\n    return x\n\n# import collections library which has functions to compute frequency of elements\nimport collections","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"85e95715b9353c353ecbc079be3c2907d8676315"},"cell_type":"code","source":"# Function to take in a DataFrame of Quora entries and output a DataFrame of swear words and frequency count\ndef count_frequency(df):\n    # put all the used bad words into a list\n    bw_list = flatten(df['bad_words'].tolist())\n    # count the frequency of each bad word\n    counter = collections.Counter(bw_list)\n    # re-order in terms of frequency\n    counter = counter.most_common()\n    # put into a DataFrame\n    df_out = pd.DataFrame(counter, columns = ['Word', 'Frequency'])\n    # output\n    return df_out\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca4dd18f62bf860c0a3f796d14b4b68c2a30ec33"},"cell_type":"markdown","source":"### Count frequency of swear words in sincere and insincere questions"},{"metadata":{"trusted":false,"_uuid":"d7c989b2efed26c2de856cf0645c5fa6c96b89c2"},"cell_type":"code","source":"freq_sincere = count_frequency(df_bw_sincere)\nfreq_insincere = count_frequency(df_bw_insincere)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"b861705ea68d70ac218c923e19244db23b81bf12"},"cell_type":"code","source":"freq_sincere.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e034bfa98cc2ca812dfc700ec440ff3ce7c98e47"},"cell_type":"code","source":"# Add a column for frequency normalised by the total number of questions in the set\nnum_sincere = len(train[ train['target'] == 0])\nnum_insincere = len(train[ train['target'] == 1])\nprint('Number of sincere questions:', num_sincere)\nprint('Number of insincere questions:', num_insincere)\n\nfreq_sincere['Normalised Frequency'] = freq_sincere['Frequency']/num_sincere\nfreq_insincere['Normalised Frequency'] = freq_insincere['Frequency']/num_insincere","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"76d5735ef980ad1862cad32cf66ddce0a56a5c93"},"cell_type":"code","source":"freq_insincere.head()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"900a27db30bfd9a18aa45be27d0bb267e80b78d5"},"cell_type":"code","source":"# Bar plot: 10 most frequent bad words in sincere questions\nsns.barplot(x='Word',\n            y='Normalised Frequency',\n            data=freq_sincere.iloc[:10]\n           ).set_title('Most frequent \\'bad\\' words in Quora sincere questions');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"822aca8e77215d2d58d68adc0624fe749517fdcc"},"cell_type":"markdown","source":"From this we can see that there are many words in the bad_words list that may be appropriate in the right context. Hence the large number of sincere Quora posts with supposed 'bad words' in them."},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"143a84683edb731aa20fc5ffdb2c8375593247e7"},"cell_type":"code","source":"# Bar plot: 10 most frequent bad words in insincere questions\nsns.barplot(x='Word',\n            y='Normalised Frequency',\n            data=freq_insincere.iloc[:10]\n           ).set_title('Most frequent \\'bad\\' words in Quora insincere questions');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a602c658d8c3679d569fe1be537b4f7088c668eb"},"cell_type":"markdown","source":"The bad words "},{"metadata":{"_uuid":"ccb3537ee8cc1bd65e1d349ee6bdb618e1dfa0e1"},"cell_type":"markdown","source":"### Scatter plot of bad words against their relative frequency in the Quora posts"},{"metadata":{"trusted":false,"_uuid":"b5df41a8ab156e09b8ce8b74cb67ca24e700d6eb"},"cell_type":"code","source":"# Create DataFrame with bad-word frequencies in sincere and insincere sets\ntemp = freq_insincere.set_index('Word')['Normalised Frequency'].rename('Frequency in insincere posts')\ntemp2 = freq_sincere.set_index('Word')['Normalised Frequency'].rename('Frequency in sincere posts')\ndf_scat = pd.concat([temp,temp2], axis=1).fillna(0)\ndf_scat.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9c4addae3b08b98b7cdf0e8ec1f5f11750fca51d"},"cell_type":"code","source":"np.arange(1,10,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b9fd076126102135f9d7d45655d5d7648aacac69"},"cell_type":"code","source":"# Scatter plot and line y=x\nax = sns.scatterplot(x='Frequency in sincere posts', \n                y='Frequency in insincere posts', \n                data=df_scat)\nax.set_xlim(left=0, right=0.002)\nax.set_ylim(bottom=0, top=0.02)\nax.set_title('''Scatter plot of bad words against their \nnormalised frequencies in the Quora posts''');\nsns.lineplot(np.linspace(0,0.002,100), 2*np.linspace(0,0.002,100),\n            color='coral')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4407b56feae48abcd8aa13e57451177a8b2f30a"},"cell_type":"markdown","source":"## Removal of certain bad words from the list\n\nLet's try removing words from the 'bad words' list, that lie below the orange line (y=x) in the scatter plot above i.e. any bad word that occurs more often in the sincere set than the insincere set"},{"metadata":{"trusted":false,"_uuid":"891ce8e6e0e80ab526f57d56690e6f72b0ac17a7"},"cell_type":"code","source":"# Extract words below the orange line\nbools = df_scat['Frequency in insincere posts'] < df_scat['Frequency in sincere posts']\nbw_remove = df_scat[bools].index.values.tolist()\n# Sample of these words\nlen(bw_remove)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"d36d02f8da4033371fcf0cbbb2880b104a7eb1ef"},"cell_type":"code","source":"# Remove any instances in train of bad words in bw_remove\ntrain_dropbw = train\nfor i in tqdm(range(len(train))):\n    entry = train_dropbw['bad_words'].iloc[i]\n    if entry != []:    \n        entry = [x for x in entry if x not in bw_remove]\n        train_dropbw['bad_words'].iloc[i] = entry\n               ","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"37d8f819221649971fe5c846ead44f812a929451"},"cell_type":"code","source":"train_dropbw.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"0fc8f4f3d15a9a4afc9dc866a25f2967be8328d8"},"cell_type":"code","source":"# Collect all entries that contain bad words\nbools_bw_nonempty = [train_dropbw['bad_words'].iloc[i] != [] for i in tqdm(range(train.shape[0]))]\ndf_bw = train_dropbw[bools_bw_nonempty]\n\n# Sincere set\ndf_bw_sincere = df_bw[df_bw['target'] == 0]\n# Insincere set\ndf_bw_insincere = df_bw[df_bw['target'] ==1 ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"979012c7f86d0a92cb715370200dcea3fdd01cb3"},"cell_type":"code","source":"bw_total_sincere = len(df_bw_sincere)\nbw_total_insincere = len(df_bw_insincere)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5df0587675ea5763c2ef11e8882581a2b78743a1"},"cell_type":"code","source":"# Proportions\nbw_ratio_sincere = bw_total_sincere/num_sincere\nbw_ratio_insincere = bw_total_insincere/num_insincere\n\n# Plot\nratio_data = pd.DataFrame(\n    {'Flag': ['Sincere', 'Insincere'],\n     'Proportion': [bw_ratio_sincere, bw_ratio_insincere]}\n)\nratio_data","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"5aed96aa0bd52aa009e1e602072b72c7f777e007"},"cell_type":"code","source":"# Plot of proportion of questions that contain bad words\nsns.barplot(x='Flag', \n            y='Proportion',\n            data=ratio_data).set_title('Proportion of entries with at least one bad word');","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8a6b6c9f1bf13cec17af4f9ad206051e9ce81681"},"cell_type":"code","source":"print('''From the training data, insincere posts are about\n{:.1f} times more likely to contain bad words'''.format(bw_ratio_insincere/bw_ratio_sincere))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22e248e0051e867c6a64abb29ee195510e299ae9"},"cell_type":"markdown","source":"## Conclusions\n\n* In the training dataset, insincere posts are 6.4 times more likely to contain a 'bad word' than sincere posts\n* Removing certain bad words that occur frequently in the sincere dataset can improve accuracy\n* Processing bad words could help predict insincere posts\n"},{"metadata":{"trusted":false,"_uuid":"8c1615e98f614e4cc8ebb376c41367fd68ffb95a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"}},"nbformat":4,"nbformat_minor":1}