{"cells":[{"metadata":{"_uuid":"81bba429462460ef1a181627b70d0b57ebbda676"},"cell_type":"markdown","source":"## Why  Feature Engineering ?\nIt might be obvious that model learns the patterns and interactions while it trains. So, how does the engineering comes into the place. Also, there is a line where we should leave some interactions for models to find out and not hand label features. The need for feature engineering;\n\n![Image](https://www.analyticsindiamag.com/wp-content/uploads/2018/01/data-cleaning.png)\n\n<sup>Source:  analyticsindiamag </sup>\n\n1.  Helps the model to catch those exceptions when you engineer a significant interaction\n2. The model converges faster if you happen to find good set of features\n3. When you have new source of information. A chance to make better model. You engineer features!\n\nLet's start with basic imports and loading the dataset...\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport json\nimport string\nimport numpy as np\nimport pandas as pd\nimport plotly.offline as py\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.graph_objs as go\n\nfrom pandas.io.json import json_normalize\nfrom plotly import tools\npy.init_notebook_mode(connected=True)\npd.options.mode.chained_assignment = None\npd.options.display.max_columns = 999\ncolor = sns.color_palette()\nnp.random.seed(13)\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3387ddbe278467542f69af3eb6fad51f4c1bd584"},"cell_type":"markdown","source":"## Pause And Think! \n\nThis is one of the stategies I adopt. I avoid looking into the head frame directly and instead focus on the competition problem. One of the reasons doing so is keeping my mind in a calm state where I could imagine what features I could think of or what insicere comments would read like?\n\n![Ask](http://www.gregorypouy.com/wp-content/uploads/2015/12/what-how-why.jpg)\n\n\n\nFew pointers I have is;\n1. Are they toxic with targetted words on race/region/religion? \n2. Do they contain obscene words ? Are these questions long or short?\n3. And, dividing them into clusters will help my model predict - what cluster of insincerity does an insincere question lies in .... etc\n\nSo, let's check the head frame now..\n"},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"837f5470c2f33072e75bd29d65ffb525ef221cdd","_kg_hide-output":false},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\nprint(\"Train shape : \", train_df.shape)\nprint(\"Test shape : \", test_df.shape)\ntrain_df.head(8)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d73d4a78376d1f3c61c2b750549fd2fdd3b7016"},"cell_type":"markdown","source":"Let's print  insincere comments and have a look at few examples to get ideas for feature engineering"},{"metadata":{"trusted":true,"_uuid":"3e252821ec139fd89c99fdd66d973c1e613d09e3"},"cell_type":"code","source":"sincere_questions = train_df[train_df['target'] == 0]\ninsincere_questions = train_df[train_df['target'] == 1]\ninsincere_questions.tail(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28b2a99641f6d317d3674134b78400d04cfd0174"},"cell_type":"markdown","source":"Now, that we have had a look. Let's try to create features for our question. "},{"metadata":{"_uuid":"fa2aef4d1e77f3323c2128a887cba6ae942e5cb4"},"cell_type":"markdown","source":"## Average Length of Sincere vs. Insincere"},{"metadata":{"trusted":true,"_uuid":"b8c7b13a5fec4c637ccb1ead135597f5413f4c67"},"cell_type":"code","source":"sincere_questions['length'] = sincere_questions['question_text'].apply(lambda x : len(x))\ninsincere_questions['length'] = insincere_questions['question_text'].apply(lambda x : len(x))\ntrain_df['length'] = train_df['question_text'].apply(lambda x : len(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6518d85b1d6894acbabd37909bfe7bfa6949224d"},"cell_type":"code","source":"sincere = go.Box(y=sincere_questions['length'].values, name = 'Sincere Questions', boxmean=True)\ninsincere = go.Box(y=insincere_questions['length'].values, name = 'Insincere Questions', boxmean=True)\ndata = [sincere, insincere]\nlayout = go.Layout(title = \"Average Length of Sincere vs. Insincere\")\nfig = go.Figure(data=data,layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef748c30d7b633290b37b6f5bdb8122a3ebaab22"},"cell_type":"markdown","source":"Seems like length doesn't explain insincerity but certainly has some information to add to the context. I am curious about the highest lenght question lets check it!"},{"metadata":{"trusted":true,"_uuid":"857494671851c9dec47461177ae858bf72560bcb"},"cell_type":"code","source":"print(train_df.iloc[443216]['question_text'])\ntrain_df[train_df['length'] == train_df['length'].max()]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a81d76bb260d9a9c8b6d717a47acdcd0161ba486"},"cell_type":"markdown","source":"Oh, looks like someone was in dire need of completing their math homework. But, its surprising to see it marked as insincere. However, its important to keep in mind that there is significant noise in our data\n\nFrom [data page](http://https://www.kaggle.com/c/quora-insincere-questions-classification/data);\n\n> Note that the distribution of questions in the dataset should not be taken to be representative of the distribution of questions asked on Quora. This is, in part, because of the combination of sampling procedures and sanitization measures that have been applied to the final dataset.\n\n"},{"metadata":{"_uuid":"91491046f84d602317991dddf585b7889d3b6c6e"},"cell_type":"markdown","source":"Now, let us create bulk of these common features; \n"},{"metadata":{"_uuid":"983a478fa34775c436f13169a441ebabda1a317c"},"cell_type":"markdown","source":"## The Meta Features Based On Word/Character"},{"metadata":{"trusted":true,"_uuid":"de0cd71c4bd6516ad115e9b9111f1b445d64f86b"},"cell_type":"code","source":"from tqdm import tqdm # I love this handy tool! \nprint(\">> Generating Count Based And Demographical Features\")\nfor df in ([train_df]):\n    df['length'] = df['question_text'].apply(lambda x : len(x))\n    df['capitals'] = df['question_text'].apply(lambda comment: sum(1 for c in comment if c.isupper()))\n    df['caps_vs_length'] = df.apply(lambda row: float(row['capitals'])/float(row['length']),axis=1)\n    df['num_exclamation_marks'] = df['question_text'].apply(lambda comment: comment.count('!'))\n    df['num_question_marks'] = df['question_text'].apply(lambda comment: comment.count('?'))\n    df['num_punctuation'] = df['question_text'].apply(lambda comment: sum(comment.count(w) for w in '.,;:'))\n    df['num_symbols'] = df['question_text'].apply(lambda comment: sum(comment.count(w) for w in '*&$%'))\n    df['num_words'] = df['question_text'].apply(lambda comment: len(comment.split()))\n    df['num_unique_words'] = df['question_text'].apply(lambda comment: len(set(w for w in comment.split())))\n    df['words_vs_unique'] = df['num_unique_words'] / df['num_words']\n    df['num_smilies'] = df['question_text'].apply(lambda comment: sum(comment.count(w) for w in (':-)', ':)', ';-)', ';)')))\n    df['num_sad'] = df['question_text'].apply(lambda comment: sum(comment.count(w) for w in (':-<', ':()', ';-()', ';(')))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4c9dbdd880d4d5379f00d88307e2c8a309c54781"},"cell_type":"markdown","source":"Let's have a glance at new features ... "},{"metadata":{"trusted":true,"_uuid":"9f8105b472c8e088fdcde1ef36fe52697c9a3ad6"},"cell_type":"code","source":"train_df[train_df.columns[2:]].head(8)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f980cb38168fcdfcb2840ebd33fa09bdcfdbc34"},"cell_type":"markdown","source":" ## Oh my! A Bad Word\n\nWe found a significant bad words in the insincere comments. Here I am using few profanity words list Let's device and see count for each and ratio features for the same. "},{"metadata":{"trusted":true,"_uuid":"a83457935b4ee5c3689f9da864246fd9a386d1fb","_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"# List Of Bad Words by Google-Profanity Words \nbad_words = ['cockknocker', 'n1gger', 'ing', 'fukker', 'nympho', 'fcuking', 'gook', 'freex', 'arschloch', 'fistfucked', 'chinc', 'raunch', 'fellatio', 'splooge', 'nutsack', 'lmfao', 'wigger', 'bastard', 'asses', 'fistfuckings', 'blue', 'waffle', 'beeyotch', 'pissin', 'dominatrix', 'fisting', 'vullva', 'paki', 'cyberfucker', 'chuj', 'penuus', 'masturbate', 'b00b*', 'fuks', 'sucked', 'fuckingshitmotherfucker', 'feces', 'panty', 'coital', 'wh00r.', 'whore', 'condom', 'hells', 'foreskin', 'wanker', 'hoer', 'sh1tz', 'shittings', 'wtf', 'recktum', 'dick*', 'pr0n', 'pasty', 'spik', 'phukked', 'assfuck', 'xxx', 'nigger*', 'ugly', 's_h_i_t', 'mamhoon', 'pornos', 'masterbates', 'mothafucks', 'Mother', 'Fukkah', 'chink', 'pussy', 'palace', 'azazel', 'fistfucking', 'ass-fucker', 'shag', 'chincs', 'duche', 'orgies', 'vag1na', 'molest', 'bollock', 'a-hole', 'seduce', 'Cock*', 'dog-fucker', 'shitz', 'Mother', 'Fucker', 'penial', 'biatch', 'junky', 'orifice', '5hit', 'kunilingus', 'cuntbag', 'hump', 'butt', 'fuck', 'titwank', 'schaffer', 'cracker', 'f.u.c.k', 'breasts', 'd1ld0', 'polac', 'boobs', 'ritard', 'fuckup', 'rape', 'hard', 'on', 'skanks', 'coksucka', 'cl1t', 'herpy', 's.o.b.', 'Motha', 'Fucker', 'penus', 'Fukker', 'p.u.s.s.y.', 'faggitt', 'b!tch', 'doosh', 'titty', 'pr1k', 'r-tard', 'gigolo', 'perse', 'lezzies', 'bollock*', 'pedophiliac', 'Ass', 'Monkey', 'mothafucker', 'amcik', 'b*tch', 'beaner', 'masterbat*', 'fucka', 'phuk', 'menses', 'pedophile', 'climax', 'cocksucking', 'fingerfucked', 'asswhole', 'basterdz', 'cahone', 'ahole', 'dickflipper', 'diligaf', 'Lesbian', 'sperm', 'pisser', 'dykes', 'Skanky', 'puuker', 'gtfo', 'orgasim', 'd0ng', 'testicle*', 'pen1s', 'piss-off', '@$$', 'fuck', 'trophy', 'arse*', 'fag', 'organ', 'potty', 'queerz', 'fannybandit', 'muthafuckaz', 'booger', 'pussypounder', 'titt', 'fuckoff', 'bootee', 'schlong', 'spunk', 'rumprammer', 'weed', 'bi7ch', 'pusse', 'blow', 'job', 'kusi*', 'assbanged', 'dumbass', 'kunts', 'chraa', 'cock', 'sucker', 'l3i+ch', 'cabron', 'arrse', 'cnut', 'how', 'to', 'murdep', 'fcuk', 'phuked', 'gang-bang', 'kuksuger', 'mothafuckers', 'ghey', 'clit', 'licker', 'feg', 'ma5terbate', 'd0uche', 'pcp', 'ejaculate', 'nigur', 'clits', 'd0uch3', 'b00bs', 'fucked', 'assbang', 'mutha', 'goddamned', 'cazzo', 'lmao', 'godamn', 'kill', 'coon', 'penis-breath', 'kyke', 'heshe', 'homo', 'tawdry', 'pissing', 'cumshot', 'motherfucker', 'menstruation', 'n1gr', 'rectus', 'oral', 'twats', 'scrot', 'God', 'damn', 'jerk', 'nigga', 'motherfuckin', 'kawk', 'homey', 'hooters', 'rump', 'dickheads', 'scrud', 'fist', 'fuck', 'carpet', 'muncher', 'cipa', 'cocaine', 'fanyy', 'frigga', 'massa', '5h1t', 'brassiere', 'inbred', 'spooge', 'shitface', 'tush', 'Fuken', 'boiolas', 'fuckass', 'wop*', 'cuntlick', 'fucker', 'bodily', 'bullshits', 'hom0', 'sumofabiatch', 'jackass', 'dilld0', 'puuke', 'cums', 'pakie', 'cock-sucker', 'pubic', 'pron', 'puta', 'penas', 'weiner', 'vaj1na', 'mthrfucker', 'souse', 'loin', 'clitoris', 'f.ck', 'dickface', 'rectal', 'whored', 'bookie', 'chota', 'bags', 'sh!t', 'pornography', 'spick', 'seamen', 'Phukker', 'beef', 'curtain', 'eat', 'hair', 'pie', 'mother', 'fucker', 'faigt', 'yeasty', 'Clit', 'kraut', 'CockSucker', 'Ekrem*', 'screwing', 'scrote', 'fubar', 'knob', 'end', 'sleazy', 'dickwhipper', 'ass', 'fuck', 'fellate', 'lesbos', 'nobjokey', 'dogging', 'fuck', 'hole', 'hymen', 'damn', 'dego', 'sphencter', 'queef*', 'gaylord', 'va1jina', 'a55', 'fuck', 'douchebag', 'blowjob', 'mibun', 'fucking', 'dago', 'heroin', 'tw4t', 'raper', 'muff', 'fitt*', 'wetback*', 'mo-fo', 'fuk*', 'klootzak', 'sux', 'damnit', 'pimmel', 'assh0lez', 'cntz', 'fux', 'gonads', 'bullshit', 'nigg3r', 'fack', 'weewee', 'shi+', 'shithead', 'pecker', 'Shytty', 'wh0re', 'a2m', 'kkk', 'penetration', 'kike', 'naked', 'kooch', 'ejaculation', 'bang', 'hoare', 'jap', 'foad', 'queef', 'buttwipe', 'Shity', 'dildo', 'dickripper', 'crackwhore', 'beaver', 'kum', 'sh!+', 'qweers', 'cocksuka', 'sexy', 'masterbating', 'peeenus', 'gays', 'cocksucks', 'b17ch', 'nad', 'j3rk0ff', 'fannyflaps', 'God-damned', 'masterbate', 'erotic', 'sadism', 'turd', 'flipping', 'the', 'bird', 'schizo', 'whiz', 'fagg1t', 'cop', 'some', 'wood', 'banger', 'Shyty', 'f', 'you', 'scag', 'soused', 'scank', 'clitorus', 'kumming', 'quim', 'penis', 'bestial', 'bimbo', 'gfy', 'spiks', 'shitings', 'phuking', 'paddy', 'mulkku', 'anal', 'leakage', 'bestiality', 'smegma', 'bull', 'shit', 'pillu*', 'schmuck', 'cuntsicle', 'fistfucker', 'shitdick', 'dirsa', 'm0f0']\nprint(\">> Words in bad_word list:\", len(bad_words))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f155f193f1ad46c6d56311316750b4b8193f915b"},"cell_type":"code","source":"print(\">> Generating Features on Bad Words\")\nfor df in ([train_df]):\n    df[\"badwordcount\"] = df['question_text'].apply(lambda comment: sum(comment.count(w) for w in bad_words))\n    df['num_chars'] =    df['question_text'].apply(len)\n    df[\"normchar_badwords\"] = df[\"badwordcount\"]/df['num_chars']\n    df[\"normword_badwords\"] = df[\"badwordcount\"]/df['num_words']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2474a42b7d0220d495eb25298202b51d46c0646d"},"cell_type":"markdown","source":"Checking the bad_word features..."},{"metadata":{"trusted":true,"_uuid":"e8a43a58d00cde59527fa99fc5e7d6d577a68bf8"},"cell_type":"code","source":"train_df[['badwordcount','num_chars','normchar_badwords','normword_badwords']].head(8)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"311a7ceed117d4ef38b1bd40e813295bdd0fe4eb"},"cell_type":"markdown","source":"## Tagging Parts Of Speech And More Feature Engineering..\n\nI suspect that the insincere questions have significant adverbs/adjective that makes them toxic. I am hopeful that these features might model understand various POS structures in the question_text\n\n\n![POS](https://cdn-images-1.medium.com/max/1600/1*fRjvBbgzo90x0MZdXZT82A.png)"},{"metadata":{"trusted":true,"_uuid":"162ae29a143dd0f0587d939c3ce96a896992110c"},"cell_type":"code","source":"import string\nfrom nltk import pos_tag\nfrom nltk.corpus import stopwords\nfrom nltk.tokenize import TweetTokenizer\n\ndef tag_part_of_speech(text):\n    text_splited = text.split(' ')\n    text_splited = [''.join(c for c in s if c not in string.punctuation) for s in text_splited]\n    text_splited = [s for s in text_splited if s]\n    pos_list = pos_tag(text_splited)\n    noun_count = len([w for w in pos_list if w[1] in ('NN','NNP','NNPS','NNS')])\n    adjective_count = len([w for w in pos_list if w[1] in ('JJ','JJR','JJS')])\n    verb_count = len([w for w in pos_list if w[1] in ('VB','VBD','VBG','VBN','VBP','VBZ')])\n    return[noun_count, adjective_count, verb_count]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ef2ed351b2fc77a4a7684e427c57807cee8591a0"},"cell_type":"code","source":"print(\">> Generating POS Features\")\nfor df in ([train_df]):\n    df['nouns'], df['adjectives'], df['verbs'] = zip(*df['question_text'].apply(\n        lambda comment: tag_part_of_speech(comment)))\n    df['nouns_vs_length'] = df['nouns'] / df['length']\n    df['adjectives_vs_length'] = df['adjectives'] / df['length']\n    df['verbs_vs_length'] = df['verbs'] /df['length']\n    df['nouns_vs_words'] = df['nouns'] / df['num_words']\n    df['adjectives_vs_words'] = df['adjectives'] / df['num_words']\n    df['verbs_vs_words'] = df['verbs'] / df['num_words']\n    # More Handy Features\n    df[\"count_words_title\"] = df[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\n    df[\"mean_word_len\"] = df[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))\n    df['punct_percent']= df['num_punctuation']*100/df['num_words']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c9fb04a5eb2e796e279cb75083ccee57684da5f"},"cell_type":"code","source":"train_df[['nouns','nouns_vs_length','adjectives_vs_length','verbs_vs_length','nouns_vs_words','adjectives_vs_words','verbs_vs_words']].head(8)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ddcbd41286b9cd422e7d80372b25f8654e31ebe8"},"cell_type":"markdown","source":"## Correlation Matrix"},{"metadata":{"trusted":true,"_uuid":"8cd7d57bf02a4ecc950f0d7e28410832fa2e5843"},"cell_type":"code","source":"f, ax = plt.subplots(figsize= [20,15])\nsns.heatmap(train_df.drop(['qid','question_text'], axis=1).corr(), annot=True, fmt=\".2f\", ax=ax, \n            cbar_kws={'label': 'Correlation Coefficient'}, cmap='viridis')\nax.set_title(\"Correlation Matrix for Insincerity and New Features\", fontsize=18)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4400e26a6c17aba702cc2b49290fa02949ba225"},"cell_type":"markdown","source":"## Feedback And More...\n* Highly appreciate your feedback. \n* Do share your feature engineering ideas and I shall see them implement in upcoming versions of this kernel\n\n\n### Todo \n* Show tutorial to use meta-features in XGBoost And LSTM Models\n* More Features And Engineering..."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}