{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Visualize Sincere vs Insincere words\n\nThis is my first kernel and i decided to visualize common words that appear in sincere sentences vs insincere sentences."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport pandas as pd\n\nfrom nltk.corpus import stopwords\nfrom wordcloud import WordCloud","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"399e5a9efe80e99aeecf741ddb2a81adcdac773d"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1844784e8385166e25ed41df96c50eed292ffe0d"},"cell_type":"markdown","source":"### Basic Stats\n\nThere are around 1.3 million questions and only 80k insincere ones, so the training dataset is very unbalanced."},{"metadata":{"trusted":true,"_uuid":"5da76099e45dfdc8895aa120fa905b0711a6677e"},"cell_type":"code","source":"print('total', train.shape[0])\nprint('sincere questions', train[train['target'] == 0].shape[0])\nprint('insincere questions', train[train['target'] == 1].shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"531649e79fca74b26cb419e9fd0b73eebaf5f51d"},"cell_type":"markdown","source":"### Vocabulary\n\nWe could just use one of the vectorizers and get the words used in the sincere and insincere sentences but i just decided to write my own. It counts the words in a sentence and builds a dict of word -> count. The tokenizer itself is a very simple split by space. \n\nWe remove stopwords because they are common for both sincere as well as insincere words."},{"metadata":{"trusted":true,"_uuid":"404f58ffe2e5ce35057463c7496229b72239242e"},"cell_type":"code","source":"class Vocabulary(object):\n    \n    def __init__(self):\n        self.vocab = {}\n        self.STOPWORDS = set()\n        self.STOPWORDS = set(stopwords.words('english'))\n        \n    def build_vocab(self, lines):\n        for line in lines:\n            for word in line.split(' '):\n                word = word.lower()\n                if (word in self.STOPWORDS):\n                    continue\n                if (word not in self.vocab):\n                    self.vocab[word] = 0\n                self.vocab[word] +=1 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bb8bbb6587bfbf5a9ecb0dcba632a862e25a42b4"},"cell_type":"code","source":"sincere_vocab = Vocabulary()\nsincere_vocab.build_vocab(train[train['target'] == 0]['question_text'])\nsincere_vocabulary = sorted(sincere_vocab.vocab.items(), reverse=True, key=lambda kv: kv[1])\nfor word, count in sincere_vocabulary[:10]:\n    print(word, count)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"feead7d6d1e803c56553d291d7c2af2631d301d3"},"cell_type":"code","source":"insincere_vocab = Vocabulary()\ninsincere_vocab.build_vocab(train[train['target'] == 1]['question_text'])\ninsincere_vocabulary = sorted(insincere_vocab.vocab.items(), reverse=True, key=lambda kv: kv[1])\nfor word, count in insincere_vocabulary[:10]:\n    print(word, count)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"936bea678d9293bb921933d7d46ead4acbcc7db4"},"cell_type":"markdown","source":"As we can clearly see there are certain words that count high in both sincere as well as insincere sentences and they are not in the stopword list. \n\nA simple metric to remove these is to use the ratio of sincere and insincere words and vice versa as the score for the word."},{"metadata":{"trusted":true,"_uuid":"d4a67b36d5ac2c102b06f7bca89d454129671752"},"cell_type":"code","source":"sincere_score = {}\nfor word, count in sincere_vocabulary:\n    sincere_score[word] = count / insincere_vocab.vocab.get(word, 1)\n\nwordcloud_sincere = WordCloud(width = 800, height = 800,background_color ='white', min_font_size = 10)\nwordcloud_sincere.generate_from_frequencies(sincere_score) \n  \nplt.figure(figsize = (8, 8), facecolor = None) \nplt.imshow(wordcloud_sincere) \nplt.axis(\"off\") \nplt.tight_layout(pad = 0) \n  \nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2bcc6ce19a09923ef8b7caff74c4aa4e93028dcd"},"cell_type":"code","source":"insincere_score = {}\nfor word, count in insincere_vocabulary:\n    insincere_score[word] = count / sincere_vocab.vocab.get(word, 1)\n\nwordcloud_insincere = WordCloud(width = 800, height = 800,background_color ='white', min_font_size = 10)\nwordcloud_insincere.generate_from_frequencies(insincere_score) \n  \nplt.figure(figsize = (8, 8), facecolor = None) \nplt.imshow(wordcloud_insincere) \nplt.axis(\"off\") \nplt.tight_layout(pad = 0) \n  \nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b0b86f87d9f2faaa62021d4bcd6c9bee40809e8"},"cell_type":"markdown","source":"## Conclusion\nAs we can clearly see there are certain words (swear words, discriminatory words based on race, political figures etc) that show up a lot in insincere sentences."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}