{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Quora Insincere Question Classification\n\n## A Kaggle challenge hosted by Quora in 2019\n\nQuora is an online Q&A platform that empowers people to learn from one another. On Quora, people can ask questions, answer questions previosuly asked or simply read others' answers. This allows people to connect online and gain unique insights.\n\nLike any other online platform, Quora also has to handle \"toxic and divisive content\". A key aspect in the fight against such content is to weed out insincere questions. This competition focuses on that particular problem and requires us to come up with a model that flags and identifies insincere questions. This will help Quora fight abusive and inappropriate content online and uphold their policy of \"Be Nice, Be Respectful\"."},{"metadata":{},"cell_type":"markdown","source":"### Problem Statement\nBuild a model that will be predicting whether a question asked on Quora is sincere or not.\n\n"},{"metadata":{},"cell_type":"markdown","source":"### Exploratory Data Analysis\n\nDescription of insincere question given by Quora :\n> An insincere question is defined as a question intended to make a statement rather than look for helpful answers. Some characteristics that can signify that a question is insincere:\n> + Has a non-neutral tone\n>   + Has an exaggerated tone to underscore a point about a group of people\n>   + Is rhetorical and meant to imply a statement about a group of people\n> * Is disparaging or inflammatory\n>   + Suggests a discriminatory idea against a protected class of people, or seeks confirmation of a stereotype\n>   + Makes disparaging attacks/insults against a specific person or group of people\n>   + Based on an outlandish premise about a group of people\n>   + Disparages against a characteristic that is not fixable and not measurable\n> + Isn't grounded in reality\n>   + Based on false information, or contains absurd assumptions\n> + Uses sexual content (incest, bestiality, pedophilia) for shock value, and not to seek genuine answers\n>\n> The training data includes the question that was asked, and whether it was identified as insincere (target = 1). The ground-truth labels contain some amount of >noise: they are not guaranteed to be perfect.\n>\n>Note that the distribution of questions in the dataset should not be taken to be representative of the distribution of questions asked on Quora. This is, in part, because of the combination of sampling procedures and sanitization measures that have been applied to the final dataset."},{"metadata":{},"cell_type":"markdown","source":"We are going to start off by playing around with the data given to us by Quora\n\nFirst, we will import the python packages that we will be needing during the course of solving this problem"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n\n%matplotlib inline\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport os\nimport csv\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport string\nimport re\nimport nltk\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\nfrom wordcloud import WordCloud","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can list the data files given by Quora."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here is a short descrpition of the data files:\n+ `train.csv` is the file containing the training set. It has three columns: \n    + `qid` - unique question identifier\n    + `question_text` - Quora question text\n    + `target` - a question labeled \"insincere\" has a value of 1, otherwise 0\n+ `test.csv` is the file containing the test set. We will run our final model on this set and use the output as our competttion submission. It has only two columns `qid` and `question_text`. \n+ `sample_submissions.csv` as the name suggests is a sample of how our submission file should look.\n+ `embeddings.zip` - As external data sources are not allowed, Quora provides 4 different embedding files that can be used in the models. They are as follows:\n    + GoogleNews-vectors-negative300 - https://code.google.com/archive/p/word2vec/\n    + glove.840B.300d - https://nlp.stanford.edu/projects/glove/\n    + paragram_300_sl999 - https://cogcomp.org/page/resource_view/106\n    + wiki-news-300d-1M - https://fasttext.cc/docs/en/english-vectors.html"},{"metadata":{},"cell_type":"markdown","source":"We will load the training and test datasets as pandas dataframes"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv('/kaggle/input/train.csv')\ntest_data = pd.read_csv('/kaggle/input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"See the first few rows of the training and test dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets see the dimensions of training set and test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Dimensions of Training Dataset : \", train_data.shape)\nprint(\"Dimensions of Test Dataset : \", test_data.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets visualise the training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(x=\"target\", data=train_data, palette=\"Set1\")\nplt.title('Target Count')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_questions = len(train_data['qid'])\nnum_sincere_questions = len(train_data.qid[train_data['target'] == 0])\nnum_insincere_questions = len(train_data.qid[train_data['target'] == 1])\nprint(\"Number of Sincere questions in the training set : \", num_sincere_questions)\nprint(\"Number of Insincere questions in the training set : \", num_insincere_questions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"values = [train_data[train_data['target']==0].shape[0], train_data[train_data['target']==1].shape[0]]\nlabels = ['Sincere Questions', 'Insincere Questions']\n\nplt.pie(values, labels=labels, autopct='%1.1f%%', shadow=True)\nplt.title('Target Distribution')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(num_sincere_questions/num_questions * 100, 'percent of training data questions are sincere')\nprint(num_insincere_questions/num_questions * 100, 'percent of training data questions are insincere')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check whether the training set contains any null values."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check whether the training set has any duplicates"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.duplicated(subset = [\"question_text\", \"qid\", \"target\"]).any()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A summary of all the information we found out individually can be got as follows:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"nltk.download('wordnet')\nnltk.download('punkt')\nnltk.download('stopwords')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Feature engineering\nLets extract a few features before cleaning the data:\n+ `freq_qid`: Frequency of qid\n+ `qlen`: Length of question text\n+ `n_words`: Number of words in Question\n+ `numeric_words`: Number of numeric words in Question\n+ `sp_char_words`: Number of special characters in Question\n+ `unique_words`: Number of unique words in Question\n+ `char_words`: Number of characters in Question"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['freq_qid'] = train_data.groupby('qid')['qid'].transform('count') \ntrain_data['qlen'] = train_data['question_text'].str.len() \ntrain_data['n_words'] = train_data['question_text'].apply(lambda row: len(row.split(\" \")))\ntrain_data['numeric_words'] = train_data['question_text'].apply(lambda row: sum(c.isdigit() for c in row))\ntrain_data['sp_char_words'] = train_data['question_text'].str.findall(r'[^a-zA-Z0–9 ]').str.len()\ntrain_data['char_words'] = train_data['question_text'].apply(lambda row: len(str(row)))\ntrain_data['unique_words'] = train_data['question_text'].apply(lambda row: len(set(str(row).split())))\n\ntrain_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data['freq_qid'] = test_data.groupby('qid')['qid'].transform('count') \ntest_data['qlen'] = test_data['question_text'].str.len() \ntest_data['n_words'] = test_data['question_text'].apply(lambda row: len(row.split(\" \")))\ntest_data['numeric_words'] = test_data['question_text'].apply(lambda row: sum(c.isdigit() for c in row))\ntest_data['sp_char_words'] = test_data['question_text'].str.findall(r'[^a-zA-Z0–9 ]').str.len()\ntest_data['char_words'] = test_data['question_text'].apply(lambda row: len(str(row)))\ntest_data['unique_words'] = test_data['question_text'].apply(lambda row: len(set(str(row).split())))\n\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Removing punctuation\nThe following function will remove any punctuation from our question text. A comprehensive list of punctuations is given in the `puncts` array."},{"metadata":{"trusted":true},"cell_type":"code","source":"puncts=[',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', \n        '•', '~', '@', '£', '·', '_', '{', '}', '©', '^', '®', '`', '<', '→', '°', '€', '™', '›', '♥', '←', '×', '§', '″', '′', \n        '█', '…', '“', '★', '”', '–', '●', '►', '−', '¢', '¬', '░', '¡', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', \n        '—', '‹', '─', '▒', '：', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', '¯', '♦', '¤', '▲', '¸', '⋅', '‘', '∞', \n        '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '・', '╦', '╣', '╔', '╗', '▬', '❤', '≤', '‡', '√', '◄', '━', \n        '⇒', '▶', '≥', '╝', '♡', '◊', '。', '✈', '≡', '☺', '✔', '↵', '≈', '✓', '♣', '☎', '℃', '◦', '└', '‟', '～', '！', '○', \n        '◆', '№', '♠', '▌', '✿', '▸', '⁄', '□', '❖', '✦', '．', '÷', '｜', '┃', '／', '￥', '╠', '↩', '✭', '▐', '☼', '☻', '┐', \n        '├', '«', '∼', '┌', '℉', '☮', '฿', '≦', '♬', '✧', '〉', '－', '⌂', '✖', '･', '◕', '※', '‖', '◀', '‰', '\\x97', '↺', \n        '∆', '┘', '┬', '╬', '،', '⌘', '⊂', '＞', '〈', '⎙', '？', '☠', '⇐', '▫', '∗', '∈', '≠', '♀', '♔', '˚', '℗', '┗', '＊', \n        '┼', '❀', '＆', '∩', '♂', '‿', '∑', '‣', '➜', '┛', '⇓', '☯', '⊖', '☀', '┳', '；', '∇', '⇑', '✰', '◇', '♯', '☞', '´', \n        '↔', '┏', '｡', '◘', '∂', '✌', '♭', '┣', '┴', '┓', '✨', '\\xa0', '˜', '❥', '┫', '℠', '✒', '［', '∫', '\\x93', '≧', '］', \n        '\\x94', '∀', '♛', '\\x96', '∨', '◎', '↻', '⇩', '＜', '≫', '✩', '✪', '♕', '؟', '₤', '☛', '╮', '␊', '＋', '┈', '％', \n        '╋', '▽', '⇨', '┻', '⊗', '￡', '।', '▂', '✯', '▇', '＿', '➤', '✞', '＝', '▷', '△', '◙', '▅', '✝', '∧', '␉', '☭', \n        '┊', '╯', '☾', '➔', '∴', '\\x92', '▃', '↳', '＾', '׳', '➢', '╭', '➡', '＠', '⊙', '☢', '˝', '∏', '„', '∥', '❝', '☐', \n        '▆', '╱', '⋙', '๏', '☁', '⇔', '▔', '\\x91', '➚', '◡', '╰', '\\x85', '♢', '˙', '۞', '✘', '✮', '☑', '⋆', 'ⓘ', '❒', \n        '☣', '✉', '⌊', '➠', '∣', '❑', '◢', 'ⓒ', '\\x80', '〒', '∕', '▮', '⦿', '✫', '✚', '⋯', '♩', '☂', '❞', '‗', '܂', '☜', \n        '‾', '✜', '╲', '∘', '⟩', '＼', '⟨', '·', '✗', '♚', '∅', 'ⓔ', '◣', '͡', '‛', '❦', '◠', '✄', '❄', '∃', '␣', '≪', '｢', \n        '≅', '◯', '☽', '∎', '｣', '❧', '̅', 'ⓐ', '↘', '⚓', '▣', '˘', '∪', '⇢', '✍', '⊥', '＃', '⎯', '↠', '۩', '☰', '◥', \n        '⊆', '✽', '⚡', '↪', '❁', '☹', '◼', '☃', '◤', '❏', 'ⓢ', '⊱', '➝', '̣', '✡', '∠', '｀', '▴', '┤', '∝', '♏', 'ⓐ', \n        '✎', ';', '␤', '＇', '❣', '✂', '✤', 'ⓞ', '☪', '✴', '⌒', '˛', '♒', '＄', '✶', '▻', 'ⓔ', '◌', '◈', '❚', '❂', '￦', \n        '◉', '╜', '̃', '✱', '╖', '❉', 'ⓡ', '↗', 'ⓣ', '♻', '➽', '׀', '✲', '✬', '☉', '▉', '≒', '☥', '⌐', '♨', '✕', 'ⓝ', \n        '⊰', '❘', '＂', '⇧', '̵', '➪', '▁', '▏', '⊃', 'ⓛ', '‚', '♰', '́', '✏', '⏑', '̶', 'ⓢ', '⩾', '￠', '❍', '≃', '⋰', '♋', \n        '､', '̂', '❋', '✳', 'ⓤ', '╤', '▕', '⌣', '✸', '℮', '⁺', '▨', '╨', 'ⓥ', '♈', '❃', '☝', '✻', '⊇', '≻', '♘', '♞', \n        '◂', '✟', '⌠', '✠', '☚', '✥', '❊', 'ⓒ', '⌈', '❅', 'ⓡ', '♧', 'ⓞ', '▭', '❱', 'ⓣ', '∟', '☕', '♺', '∵', '⍝', 'ⓑ', \n        '✵', '✣', '٭', '♆', 'ⓘ', '∶', '⚜', '◞', '்', '✹', '➥', '↕', '̳', '∷', '✋', '➧', '∋', '̿', 'ͧ', '┅', '⥤', '⬆', '⋱', \n        '☄', '↖', '⋮', '۔', '♌', 'ⓛ', '╕', '♓', '❯', '♍', '▋', '✺', '⭐', '✾', '♊', '➣', '▿', 'ⓑ', '♉', '⏠', '◾', '▹', \n        '⩽', '↦', '╥', '⍵', '⌋', '։', '➨', '∮', '⇥', 'ⓗ', 'ⓓ', '⁻', '⎝', '⌥', '⌉', '◔', '◑', '✼', '♎', '♐', '╪', '⊚', \n        '☒', '⇤', 'ⓜ', '⎠', '◐', '⚠', '╞', '◗', '⎕', 'ⓨ', '☟', 'ⓟ', '♟', '❈', '↬', 'ⓓ', '◻', '♮', '❙', '♤', '∉', '؛', \n        '⁂', 'ⓝ', '־', '♑', '╫', '╓', '╳', '⬅', '☔', '☸', '┄', '╧', '׃', '⎢', '❆', '⋄', '⚫', '̏', '☏', '➞', '͂', '␙', \n        'ⓤ', '◟', '̊', '⚐', '✙', '↙', '̾', '℘', '✷', '⍺', '❌', '⊢', '▵', '✅', 'ⓖ', '☨', '▰', '╡', 'ⓜ', '☤', '∽', '╘', \n        '˹', '↨', '♙', '⬇', '♱', '⌡', '⠀', '╛', '❕', '┉', 'ⓟ', '̀', '♖', 'ⓚ', '┆', '⎜', '◜', '⚾', '⤴', '✇', '╟', '⎛', \n        '☩', '➲', '➟', 'ⓥ', 'ⓗ', '⏝', '◃', '╢', '↯', '✆', '˃', '⍴', '❇', '⚽', '╒', '̸', '♜', '☓', '➳', '⇄', '☬', '⚑', \n        '✐', '⌃', '◅', '▢', '❐', '∊', '☈', '॥', '⎮', '▩', 'ு', '⊹', '‵', '␔', '☊', '➸', '̌', '☿', '⇉', '⊳', '╙', 'ⓦ', \n        '⇣', '｛', '̄', '↝', '⎟', '▍', '❗', '״', '΄', '▞', '◁', '⛄', '⇝', '⎪', '♁', '⇠', '☇', '✊', 'ி', '｝', '⭕', '➘', \n        '⁀', '☙', '❛', '❓', '⟲', '⇀', '≲', 'ⓕ', '⎥', '\\u06dd', 'ͤ', '₋', '̱', '̎', '♝', '≳', '▙', '➭', '܀', 'ⓖ', '⇛', '▊', \n        '⇗', '̷', '⇱', '℅', 'ⓧ', '⚛', '̐', '̕', '⇌', '␀', '≌', 'ⓦ', '⊤', '̓', '☦', 'ⓕ', '▜', '➙', 'ⓨ', '⌨', '◮', '☷', \n        '◍', 'ⓚ', '≔', '⏩', '⍳', '℞', '┋', '˻', '▚', '≺', 'ْ', '▟', '➻', '̪', '⏪', '̉', '⎞', '┇', '⍟', '⇪', '▎', '⇦', '␝', \n        '⤷', '≖', '⟶', '♗', '̴', '♄', 'ͨ', '̈', '❜', '̡', '▛', '✁', '➩', 'ா', '˂', '↥', '⏎', '⎷', '̲', '➖', '↲', '⩵', '̗', '❢', \n        '≎', '⚔', '⇇', '̑', '⊿', '̖', '☍', '➹', '⥊', '⁁', '✢']\n\ndef clean_punct(x):\n    for punct in puncts:\n        if punct in x:\n            x = x.replace(punct, '{}' .format(punct))\n    return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Cleaning numbers\nThe following function will remove any numbers in the question text."},{"metadata":{"trusted":true},"cell_type":"code","source":"def clean_numbers(x):\n    if bool(re.search(r'\\d', x)):\n        x = re.sub('[0-9]{5,}', '#####', x)\n        x = re.sub('[0-9]{4}', '####', x)\n        x = re.sub('[0-9]{3}', '###', x)\n        x = re.sub('[0-9]{2}', '##', x)\n    return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Correcting commonly misspelled words\nThe following function will correct commonly misspelled words in the question text, so that each word is properly understood in the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"mispell_dict = {'colour': 'color', 'centre': 'center', 'favourite': 'favorite', 'travelling': 'traveling', 'counselling': 'counseling', 'theatre': 'theater', 'cancelled': 'canceled', 'labour': 'labor', 'organisation': 'organization', 'wwii': 'world war 2', 'citicise': 'criticize', 'youtu ': 'youtube ', 'Qoura': 'Quora', 'sallary': 'salary', 'Whta': 'What', 'narcisist': 'narcissist', 'howdo': 'how do', 'whatare': 'what are', 'howcan': 'how can', 'howmuch': 'how much', 'howmany': 'how many', 'whydo': 'why do', 'doI': 'do I', 'theBest': 'the best', 'howdoes': 'how does', 'mastrubation': 'masturbation', 'mastrubate': 'masturbate', \"mastrubating\": 'masturbating', 'pennis': 'penis', 'Etherium': 'bitcoin', 'narcissit': 'narcissist', 'bigdata': 'big data', '2k17': '2017', '2k18': '2018', 'qouta': 'quota', 'exboyfriend': 'ex boyfriend', 'airhostess': 'air hostess', \"whst\": 'what', 'watsapp': 'whatsapp', 'demonitisation': 'demonetization', 'demonitization': 'demonetization', 'demonetisation': 'demonetization', \n                'electroneum':'bitcoin','nanodegree':'degree','hotstar':'star','dream11':'dream','ftre':'fire','tensorflow':'framework','unocoin':'bitcoin',\n                'lnmiit':'limit','unacademy':'academy','altcoin':'bitcoin','altcoins':'bitcoin','litecoin':'bitcoin','coinbase':'bitcoin','cryptocurency':'cryptocurrency',\n                'simpliv':'simple','quoras':'quora','schizoids':'psychopath','remainers':'remainder','twinflame':'soulmate','quorans':'quora','brexit':'demonetized',\n                'iiest':'institute','dceu':'comics','pessat':'exam','uceed':'college','bhakts':'devotee','boruto':'anime',\n                'cryptocoin':'bitcoin','blockchains':'blockchain','fiancee':'fiance','redmi':'smartphone','oneplus':'smartphone','qoura':'quora','deepmind':'framework','ryzen':'cpu','whattsapp':'whatsapp',\n                'undertale':'adventure','zenfone':'smartphone','cryptocurencies':'cryptocurrencies','koinex':'bitcoin','zebpay':'bitcoin','binance':'bitcoin','whtsapp':'whatsapp',\n                'reactjs':'framework','bittrex':'bitcoin','bitconnect':'bitcoin','bitfinex':'bitcoin','yourquote':'your quote','whyis':'why is','jiophone':'smartphone',\n                'dogecoin':'bitcoin','onecoin':'bitcoin','poloniex':'bitcoin','7700k':'cpu','angular2':'framework','segwit2x':'bitcoin','hashflare':'bitcoin','940mx':'gpu',\n                'openai':'framework','hashflare':'bitcoin','1050ti':'gpu','nearbuy':'near buy','freebitco':'bitcoin','antminer':'bitcoin','filecoin':'bitcoin','whatapp':'whatsapp',\n                'empowr':'empower','1080ti':'gpu','crytocurrency':'cryptocurrency','8700k':'cpu','whatsaap':'whatsapp','g4560':'cpu','payymoney':'pay money',\n                'fuckboys':'fuck boys','intenship':'internship','zcash':'bitcoin','demonatisation':'demonetization','narcicist':'narcissist','mastuburation':'masturbation',\n                'trignometric':'trigonometric','cryptocurreny':'cryptocurrency','howdid':'how did','crytocurrencies':'cryptocurrencies','phycopath':'psychopath',\n                'bytecoin':'bitcoin','possesiveness':'possessiveness','scollege':'college','humanties':'humanities','altacoin':'bitcoin','demonitised':'demonetized',\n                'brasília':'brazilia','accolite':'accolyte','econimics':'economics','varrier':'warrier','quroa':'quora','statergy':'strategy','langague':'language',\n                'splatoon':'game','7600k':'cpu','gate2018':'gate 2018','in2018':'in 2018','narcassist':'narcissist','jiocoin':'bitcoin','hnlu':'hulu','7300hq':'cpu',\n                'weatern':'western','interledger':'blockchain','deplation':'deflation', 'cryptocurrencies':'cryptocurrency', 'bitcoin':'blockchain cryptocurrency',}\n\ndef _get_mispell(mispell_dict):\n    mispell_re = re.compile('(%s)' % '|'.join(mispell_dict.keys()))\n    return mispell_dict, mispell_re\n\nmispellings, mispellings_re = _get_mispell(mispell_dict)\ndef replace_typical_misspell(text):\n    def replace(match):\n        return mispellings[match.group(0)]\n    return mispellings_re.sub(replace, text)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Removing contractions\nThe following function will expand the English language contractions to help maintain uniformity."},{"metadata":{"trusted":true},"cell_type":"code","source":"contraction_dict = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",  \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \"you're\": \"you are\", \"you've\": \"you have\"}\n\ndef _get_contractions(contraction_dict):\n    contraction_re = re.compile('(%s)' % '|'.join(contraction_dict.keys()))\n    return contraction_dict, contraction_re\n\ncontractions, contractions_re = _get_contractions(contraction_dict)\n\ndef replace_contractions(text):\n    def replace(match):\n        return contractions[match.group(0)]\n    return contractions_re.sub(replace, text)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Removing stopwords\nThere are certain words that don't affect the understanding of the statement. The following function removes such words."},{"metadata":{"trusted":true},"cell_type":"code","source":"stopword_list = nltk.corpus.stopwords.words('english')\ndef remove_stopwords(text, is_lower_case=True):\n    tokenizer = ToktokTokenizer()\n    tokens = tokenizer.tokenize(text)\n    tokens = [token.strip() for token in tokens]\n    if is_lower_case:\n        filtered_tokens = [token for token in tokens if token not in stopword_list]\n    else:\n        filtered_tokens = [token for token in tokens if token.lower() not in stopword_list]\n    filtered_text = ' '.join(filtered_tokens)\n    return filtered_text","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Stemming\nIt gives us the root English word from plurals, gerunds, etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"from nltk.stem import  SnowballStemmer\nfrom nltk.tokenize.toktok import ToktokTokenizer\ndef stem_text(text):\n    tokenizer = ToktokTokenizer()\n    stemmer = SnowballStemmer('english')\n    tokens = tokenizer.tokenize(text)\n    tokens = [token.strip() for token in tokens]\n    tokens = [stemmer.stem(token) for token in tokens]\n    return ' '.join(tokens)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Lemmatization\nThis is similar to Stemming but works only on words present in the dictionary."},{"metadata":{"trusted":true},"cell_type":"code","source":"from nltk.stem import WordNetLemmatizer\nfrom nltk.tokenize.toktok import ToktokTokenizer\nwordnet_lemmatizer = WordNetLemmatizer()\ndef lemma_text(text):\n    tokenizer = ToktokTokenizer()\n    tokens = tokenizer.tokenize(text)\n    tokens = [token.strip() for token in tokens]\n    tokens = [wordnet_lemmatizer.lemmatize(token) for token in tokens]\n    return ' '.join(tokens)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This function is used to call all the pre-processing functions we defined earlier on the question text."},{"metadata":{"trusted":true},"cell_type":"code","source":"def clean_sentence(x):\n    x = x.lower()\n    x = clean_punct(x)\n    x = clean_numbers(x)\n    x = replace_typical_misspell(x)\n    x = remove_stopwords(x)\n    x = replace_contractions(x)\n    x = stem_text(x)\n    x = lemma_text(x)\n    x = x.replace(\"'\",\"\")\n    return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will now pre-process the question text in the training dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['preprocessed_question_text'] = train_data['question_text'].apply(lambda x: clean_sentence(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.preprocessed_question_text.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, pre-process the question text in the test dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data['preprocessed_question_text'] = test_data['question_text'].apply(lambda x: clean_sentence(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.preprocessed_question_text.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets have a look at the word clouds for the question_text before and after pre-processing for both sincere and insincere questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"def cloud(text, title, size = (10,7)):\n    # Processing Text\n    words_list = text.unique().tolist()\n    words = ' '.join(words_list)\n    \n    wordcloud = WordCloud(width=800, height=400,\n                          collocations=False\n                         ).generate(words)\n    \n    # Output Visualization\n    fig = plt.figure(figsize=size, dpi=80, facecolor='k',edgecolor='k')\n    plt.imshow(wordcloud,interpolation='bilinear')\n    plt.axis('off')\n    plt.title(title, fontsize=25,color='w')\n    plt.tight_layout(pad=0)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cloud(train_data[train_data['target']==0]['question_text'], 'Sincere Questions On question_text')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cloud(train_data[train_data['target']==0]['preprocessed_question_text'], 'Sincere Questions On preprocessed_question_text')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cloud(train_data[train_data['target']==1]['question_text'], 'Insincere Questions On question_text')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the insincere questions contain certain offensive words and most questions are related to Trump. Muslim, Women, White, sex, hate, etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"cloud(train_data[train_data['target']==1]['preprocessed_question_text'], 'Insincere Questions On preprocessed_question_text')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport time\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nimport math\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation, CuDNNGRU, Conv1D\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## split to train and val\ntrain_data, val_data = train_test_split(train_data, test_size=0.1, random_state=2018)\n\n## some config values \nembed_size = 300 # how big is each word vector\nmax_features = 50000 # how many unique words to use (i.e num rows in embedding vector)\nmaxlen = 100 # max number of words in a question to use\n\n## fill up the missing values\ntrain_X = train_data[\"question_text\"].fillna(\"_na_\").values\nval_X = val_data[\"question_text\"].fillna(\"_na_\").values\ntest_X = test_data[\"question_text\"].fillna(\"_na_\").values\n\n## Tokenize the sentences\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(train_X))\ntrain_X = tokenizer.texts_to_sequences(train_X)\nval_X = tokenizer.texts_to_sequences(val_X)\ntest_X = tokenizer.texts_to_sequences(test_X)\n\n## Pad the sentences \ntrain_X = pad_sequences(train_X, maxlen=maxlen)\nval_X = pad_sequences(val_X, maxlen=maxlen)\ntest_X = pad_sequences(test_X, maxlen=maxlen)\n\n## Get the target values\ntrain_y = train_data['target'].values\nval_y = val_data['target'].values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Bidirectional GRU"},{"metadata":{"trusted":true},"cell_type":"code","source":"inp = Input(shape=(maxlen,))\nx = Embedding(max_features, embed_size)(inp)\nx = Bidirectional(LSTM(64, return_sequences=True))(x)\nx = GlobalMaxPool1D()(x)\nx = Dense(16, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(1, activation=\"sigmoid\")(x)\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n\nprint(model.summary())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(train_X, train_y, batch_size=512, epochs=2, validation_data=(val_X, val_y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_noemb_val_y = model.predict([val_X], batch_size=1024, verbose=1)\nfor thresh in np.arange(0.1, 0.501, 0.01):\n    thresh = np.round(thresh, 2)\n    print(\"F1 score at threshold {0} is {1}\".format(thresh, metrics.f1_score(val_y, (pred_noemb_val_y>thresh).astype(int))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_noemb_test_y = model.predict([test_X], batch_size=1024, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_noemb_test_y = (pred_noemb_test_y > 0.33).astype(int)\nout_df = pd.DataFrame({\"qid\":test_data[\"qid\"].values})\nout_df['prediction'] = pred_noemb_test_y\nout_df.to_csv(\"submission.csv\", index=False)\nprint('Successfully saved submission')\npred_noemb_test_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}