{"cells":[{"metadata":{"_uuid":"1ba7e86419563c71fa8dc60685eec31c410ae220"},"cell_type":"markdown","source":"## Quora Dataset Sincere vs InSincere classification\n\nAn existential problem for any major website today is how to handle toxic and divisive content. Quora wants to tackle this problem head-on to keep their platform a place where users can feel safe sharing their knowledge with the world.\n\n#### About the Data \nTrain and test data set are provided . Test Data set doesn't have the target column.\n\nExpectation is to reuse the embeddings provided \n\n#### About the Notebook\nThis notebook covers the basic EDA on this data set The following are covered .\n1. Word count  Sincere vs Insincere\n2. Sentence length Sincere vs Insincere \n3. Vocabulary based Sentiment Sincere vs InSincere \n4. Unigram Sincere vs InSincere\n5. NER analysis of InSincere PERSON/Organisation"},{"metadata":{"trusted":true,"_uuid":"99cadca0b5bdd9cfffcf10793d7428b3a0116956"},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0d0846840b6adac48251280ea73cb654cc7d4533"},"cell_type":"code","source":"traindata=pd.read_csv('../input/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9fa15ebc59aea5c0e8bffb40c8c911f81a7c593e"},"cell_type":"code","source":"traindata.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe8048efef24d385c9d480e6bc34fae5bda2d7ad"},"cell_type":"code","source":"traindata.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a22108ed07d37e56e599f649c0405131e90be05"},"cell_type":"code","source":"traindata.target.value_counts().plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"753f985a41b6fd8d87276c6ea6d97a22850cbe10"},"cell_type":"code","source":"import nltk\nfrom nltk.tokenize import word_tokenize\ntraindata['wordlen']=traindata['question_text'].apply(lambda x: len(word_tokenize(x)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"230f80c382194d4516648792f73e957b0e1113c1"},"cell_type":"code","source":"traindata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3ee2eb44b7682cefffe77ccafe58a501d90f3b29"},"cell_type":"code","source":"sns.boxplot(x='target',y='wordlen',data=traindata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26968d8924ed15ea1e0c4fcc60792830da5abbce"},"cell_type":"code","source":"print(\"Max,Mean and Min of word count for Sincere questions\")\nprint(traindata[traindata.target==0]['wordlen'].max())\nprint(traindata[traindata.target==0]['wordlen'].mean())\nprint(traindata[traindata.target==0]['wordlen'].min())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"97be4f0d0081b57bbd52504563b62496f4721b30"},"cell_type":"code","source":"print(\"Query with maximum word count\", traindata[(traindata.target==0) & (traindata.wordlen==traindata[traindata.target==0]['wordlen'].max())]['question_text'])\n                                                 \nprint(\"Query with Minimum word count\", traindata[(traindata.target==0) & (traindata.wordlen==traindata[traindata.target==0]['wordlen'].min())]['question_text'])\n                                                 \n                                                 ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb1fe72bcb58877a596c2cb2b55f89d354f28b0d"},"cell_type":"markdown","source":"#### Let's look at the query with max and min words"},{"metadata":{"trusted":true,"_uuid":"dcfc813d9b6c561008a2c81008e3be81b3fa7ed5"},"cell_type":"code","source":"print(\"Max,Mean and Min of word count for Sincere questions\")\nprint(traindata[traindata.target==1]['wordlen'].max())\nprint(traindata[traindata.target==1]['wordlen'].mean())\nprint(traindata[traindata.target==1]['wordlen'].min())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"32551308bb4227fbb3195d6c533e7fb131a64b8c"},"cell_type":"code","source":"print(\"Query with maximum word count\", traindata[(traindata.target==1) & (traindata.wordlen==traindata[traindata.target==1]['wordlen'].max())]['question_text'])\n                                                 \nprint(\"Query with Minimum word count\", traindata[(traindata.target==1) & (traindata.wordlen==traindata[traindata.target==1]['wordlen'].min())]['question_text'])\n                                                 \n                                                 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82aae0ed7e9a400e18cb1d542e967e9582320eaa"},"cell_type":"code","source":"traindata['sentencelen']=traindata['question_text'].apply(lambda x: len(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3daa949c7826298bed434308f6199742ed9c4bf0"},"cell_type":"code","source":"print(\"Max,Mean and Min of word count for Sincere questions\")\nprint(traindata[traindata.target==0]['sentencelen'].max())\nprint(traindata[traindata.target==0]['sentencelen'].mean())\nprint(traindata[traindata.target==0]['sentencelen'].min())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa664e7ca00248add4d93dce244ef154d64984cd"},"cell_type":"code","source":"print(\"Max,Mean and Min of word count for Sincere questions\")\nprint(traindata[traindata.target==1]['sentencelen'].max())\nprint(traindata[traindata.target==1]['sentencelen'].mean())\nprint(traindata[traindata.target==1]['sentencelen'].min())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"25119eb17ee10a06720c0c1b9cbf95171211d163"},"cell_type":"code","source":"traindata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2343b6052012af379fa5a460115f81d5aad15116"},"cell_type":"code","source":"sns.boxplot(x='target',y='sentencelen',data=traindata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"593ae6d6e97c1ab2097bf111b1c9b976d384ad90"},"cell_type":"code","source":"from nltk.sentiment.vader import SentimentIntensityAnalyzer\nanalyzer = SentimentIntensityAnalyzer()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"870cbcb04779308e4667a1d882cf04d6ad682ba4"},"cell_type":"markdown","source":"### Vocabulary based sentiment Analysis"},{"metadata":{"trusted":true,"_uuid":"7fa94f9067fc063a94224b8e68c3667066f13ce5"},"cell_type":"code","source":"traindata['sentiment']=traindata['question_text'].apply(lambda x: analyzer.polarity_scores(x)['compound'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"960a6a8e1a50053b3df2ceec44c35331ceb6fedf"},"cell_type":"code","source":"sns.boxplot(x='target',y='sentiment',data=traindata)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff1f8ac2ddbbf452afb97d5499536bdb4ced5037"},"cell_type":"code","source":"sns.distplot(traindata[traindata.target==0]['sentiment'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68e3dafc7ad4ac62c3b2adc361ed13c8cde13c76"},"cell_type":"code","source":"sns.distplot(traindata[traindata.target==1]['sentiment'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a39eddc47e227a49f512779a10d552d20f2c224d"},"cell_type":"markdown","source":"### Clear indication of the sentiment is negative for target==1 and outliers for negative sentiment when target==0\n\n### This could be a good feature to feed in"},{"metadata":{"_uuid":"ef9c48930e798d3d2fbc491b2f66502e882ec715"},"cell_type":"markdown","source":"### Word Cloud"},{"metadata":{"trusted":true,"_uuid":"18202ab498f135b9960332fef6ffd334db779689"},"cell_type":"code","source":"from wordcloud import WordCloud\nfrom nltk.corpus import stopwords\nimport re","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0aa198f4de559cb263b8c93320fb253010123e36"},"cell_type":"code","source":"from tqdm import tqdm\ndef preprocess_narrative( questions ):\n    final=\"\"\n    for text in tqdm(questions):\n        text = re.sub(r'[^a-zA-Z0-9\\s]', '', text)\n        words = text.lower().split()  \n        stops = set(stopwords.words(\"english\")) \n        for w in words:\n            if w not in stops:\n                final=final+\" \"+w\n    #print(final)\n    return final\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54bb60de69f5073d1fc8cf0ec07675a7e6211cac"},"cell_type":"code","source":"x=preprocess_narrative(traindata[traindata.target==1]['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d44ca949f8592b6b9a5b8de85ad0bd58f9231af5"},"cell_type":"code","source":"wc = WordCloud(background_color=\"white\", max_words=1000,width=1000, height=500)# mask=alice_mask)\nwc.generate(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ffc285355e277bbea777863e48c6354861d43511"},"cell_type":"code","source":"fig = plt.figure(figsize = (10, 10))\nplt.imshow(wc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"944fa288ca4e34d46cbb59e95ea8236f4119e888"},"cell_type":"code","source":"# sampling the data set for 0 as the data set is huge \ntempdata=traindata[traindata.target==0]\ny=preprocess_narrative(tempdata.sample(frac=0.1)['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d765516ccf53427b494a2e95944642d5dab8e36c"},"cell_type":"code","source":"wc.generate(y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"00b832146eeae95786ee769b42927300310955e1"},"cell_type":"code","source":"fig = plt.figure(figsize = (10, 10))\nplt.imshow(wc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8836953bff8802e9b74b1de1f7c64bef76b696e6"},"cell_type":"markdown","source":"### unigram"},{"metadata":{"trusted":true,"_uuid":"a1900f7d3897c23eb1550a243321c309233e64c5"},"cell_type":"code","source":"from nltk.tokenize import WordPunctTokenizer\nfrom collections import Counter\nfrom string import punctuation, ascii_lowercase\nimport regex as re\nfrom tqdm import tqdm\n# setup tokenizer\ntokenizer = WordPunctTokenizer()\n\nstops = set(stopwords.words(\"english\"))\ndef text_to_wordlist(text, lower=False):\n    text = re.sub(r'[^a-zA-Z0-9\\s]', '', text)\n        \n    # Tokenize\n    text = tokenizer.tokenize(text)\n    \n    # optional: lower case\n    if lower:\n        text = [t.lower() for t in text]\n    \n    \n    text = [t if t not in stops else None for t in text]\n    \n    \n    \n    # Return a list of words\n    vocab.update(text)\n    #return text\n\ndef process_comments(list_sentences, lower=False):\n    comments = []\n    for text in tqdm(list_sentences):\n        text_to_wordlist(text, lower=lower)\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9836cb9744689c459da8f0e190c124ff8c939449"},"cell_type":"code","source":"vocab=Counter()\nprocess_comments(traindata[traindata.target==0]['question_text'],True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0673470d3a634a514bcdbf35c647663c59795bd8"},"cell_type":"code","source":"vocab.pop(None)\nsince_most_common=vocab.most_common(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35d5a6239ab82a8ad58b2aa4bc3475e9edf770ee"},"cell_type":"code","source":"vocab=Counter()\nprocess_comments(traindata[traindata.target==1]['question_text'],True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5511b2b60686b6403e21452faba286bda9af55d0"},"cell_type":"code","source":"vocab.pop(None)\ninsincere_most_common=vocab.most_common(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a913d77e5a14e898d1352a9ddea12ff88a4f7e7a"},"cell_type":"code","source":"sincere_mc=pd.DataFrame(since_most_common)\ninsincere_mc=pd.DataFrame(insincere_most_common)\nsincere_mc.columns=['word','count']\ninsincere_mc.columns=['word','count']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2101a0a313a4ff21809f5dc7f7a3d38eb25f7ce0"},"cell_type":"code","source":"sincere_mc.plot(x='word',kind='bar')\ninsincere_mc.plot(x='word',kind='bar')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b90c7649eff11aa6fc4a5421add318f8d8a390e3"},"cell_type":"markdown","source":"### Check the most frequent words of target==0 and sentiment is negative "},{"metadata":{"trusted":true,"_uuid":"26515169778d73a1499237d74d0ba4a9c2b34acd"},"cell_type":"code","source":"vocab=Counter()\nprocess_comments(traindata[(traindata.target==0) & (traindata.sentiment <0)]['question_text'],True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42485702289f2be443b476a0be86f6dcc594b9ff"},"cell_type":"code","source":"vocab.pop(None)\nnegative_sincere_most_common=vocab.most_common(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0532f7cbbf392135bf62cd93731e54a0f93a681d"},"cell_type":"code","source":"negative_sincere_most_common=pd.DataFrame(negative_sincere_most_common)\nnegative_sincere_most_common.columns=['word','count']\nnegative_sincere_most_common.plot(x='word',kind='bar')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5fab083052309ab0d886fa049978190f607627c3"},"cell_type":"markdown","source":"### Checking Which PERSON has been referred more in the insincere questions\n#### Trump should definitely come up , Let's see who else comes up "},{"metadata":{"trusted":true,"_uuid":"f0efb9553c72dfdb0b9d0ca3e4be409d0193b1e0"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d86eca709028dd15b9778eea6cbf8e97a89ac86"},"cell_type":"code","source":"import spacy\nfrom spacy import displacy\nfrom collections import Counter\nimport en_core_web_sm\nnlp = en_core_web_sm.load()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f00b786e5b2cd0dc7bba692045e5840e9d72fd49"},"cell_type":"code","source":"from nltk.tokenize import WordPunctTokenizer\nfrom collections import Counter\nfrom string import punctuation, ascii_lowercase\nimport regex as re\nfrom tqdm import tqdm\n# setup tokenizer\ntokenizer = WordPunctTokenizer()\nvocab=Counter()\norg=Counter()\nstops = set(stopwords.words(\"english\"))\nlabels=[]\ndef process_ner(list_sentences):\n    for text in tqdm(list_sentences):\n        \n        doc = nlp(text)\n        for x in doc.ents:\n            if(x.label_=='PERSON'):\n                vocab.update([x.text.lower()])\n            if(x.label_=='ORG'):\n                org.update([x.text.lower()])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf8e7c7139f8b5650bd71f2949196e81057aa946"},"cell_type":"markdown","source":"#### Sampling the data  50 % "},{"metadata":{"trusted":true,"_uuid":"d96aaff9047f4270da5247b7f2a929381637209e"},"cell_type":"code","source":"process_ner(traindata[traindata.target==1].sample(frac=0.5)['question_text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b55ef1c5442eb1b52a9f5118d168101b7bd070a5"},"cell_type":"code","source":"plt.figure(figsize=(20,20))\nperson_most_common=pd.DataFrame(vocab.most_common(50))\nperson_most_common.columns=['Name','count']\npersonplot=sns.barplot(y=\"count\",x=\"Name\",data=person_most_common)\nloc, labels = plt.xticks(rotation='vertical')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbae6c50ace4c2b2fc4a8be29b1b9dc5572dbe27"},"cell_type":"code","source":"plt.figure(figsize=(20,20))\norg_most_common=pd.DataFrame(org.most_common(50))\norg_most_common.columns=['Name','count']\norgplot=sns.barplot(y=\"count\",x=\"Name\",data=org_most_common)\nloc, labels = plt.xticks(rotation='vertical')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"27cef4ea62bc30357d1611839567e2e14e5980b8"},"cell_type":"markdown","source":"#### Insincere Questions seem to have religious orientation"},{"metadata":{"trusted":true,"_uuid":"ffd1386e49497533f916ab1c4a2982b8748d0916"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}