{"cells":[{"metadata":{"_uuid":"b0fc927d5c781513721cad065b8a963ebf2cb086"},"cell_type":"markdown","source":"# Text Pre-processing Techniques\nThese techniques may or may not be useful for this competition. Given the fact that is a text competition, i thought that it would be a good oportunity to present them. \n\nI have used them before in two papers. [A Comparison of Pre-processing Techniques for Twitter Sentiment Analysis](https://link.springer.com/chapter/10.1007/978-3-319-67008-9_31) and [A comparative evaluation of pre-processing techniques and their interactions for twitter sentiment analysis](https://www.sciencedirect.com/science/article/pii/S0957417418303683). \n\nThe full code is on this [Github repository](https://github.com/Deffro/text-preprocessing-techniques) with some extra techniques."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport re","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c0fc564965ba7e9a4e06e39427b0e24c83bd825"},"cell_type":"markdown","source":"## Load Dataset and print some questions"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"scrolled":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/train.csv\")\nX_train = train_df[\"question_text\"].fillna(\"dieter\").values\ntest_df = pd.read_csv(\"../input/test.csv\")\nX_test = test_df[\"question_text\"].fillna(\"dieter\").values\ny = train_df[\"target\"]\n\ntext = train_df['question_text']\n\nfor row in text[:10]:\n    print(row)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2bb57f780def82431b92aa614b51ad3b24ec69f"},"cell_type":"markdown","source":"## 1. Remove Numbers\n**Example:** Which is best powerbank for iPhone 7 in India? -> Which is best powerbank for iPhone  in India?"},{"metadata":{"trusted":true,"_uuid":"59ffcfcdd5548dba6da994f58bc9088f3e10d4bc"},"cell_type":"code","source":"def removeNumbers(text):\n    \"\"\" Removes integers \"\"\"\n    text = ''.join([i for i in text if not i.isdigit()])         \n    return text\n\ntext_removeNumbers = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_removeNumbers['TextBefore'] = text.copy()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54fa3bb88566401e54c15e40d188b0cbb25b6b51"},"cell_type":"code","source":"for index, row in text_removeNumbers.iterrows():\n    row['TextAfter'] = removeNumbers(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5048efc7bdd6bf8d9b75ea774a1804495dd5dfe"},"cell_type":"code","source":"text_removeNumbers['Changed'] = np.where(text_removeNumbers['TextBefore']==text_removeNumbers['TextAfter'], 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_removeNumbers[text_removeNumbers['Changed']=='yes']), len(text_removeNumbers), 100*len(text_removeNumbers[text_removeNumbers['Changed']=='yes'])/len(text_removeNumbers)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9148d67930f06161856bb97a5214a344ddf6e392"},"cell_type":"code","source":"for index, row in text_removeNumbers[text_removeNumbers['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a14dcfd5ce1b6c3530750a2d8874c1b0c3fcf046"},"cell_type":"markdown","source":"## 2. Replace Repetitions of Punctuation\nThis technique:\n - replaces repetitions of exlamation marks with the tag \"multiExclamation\"\n - replaces repetitions of question marks with the tag \"multiQuestion\"\n - replaces repetitions of stop marks with the tag \"multiStop\"\n \n **Example:** How do I overcome the fear of facing an interview? It's killing me inside..what should I do? -> How do I overcome the fear of facing an interview? It's killing me inside multiStop what should I do?"},{"metadata":{"trusted":true,"_uuid":"aa1fb76fd024e5ec6abe6aa289760ee4895473ef"},"cell_type":"code","source":"def replaceMultiExclamationMark(text):\n    \"\"\" Replaces repetitions of exlamation marks \"\"\"\n    text = re.sub(r\"(\\!)\\1+\", ' multiExclamation ', text)\n    return text\n\ndef replaceMultiQuestionMark(text):\n    \"\"\" Replaces repetitions of question marks \"\"\"\n    text = re.sub(r\"(\\?)\\1+\", ' multiQuestion ', text)\n    return text\n\ndef replaceMultiStopMark(text):\n    \"\"\" Replaces repetitions of stop marks \"\"\"\n    text = re.sub(r\"(\\.)\\1+\", ' multiStop ', text)\n    return text\n\ntext_replaceRepOfPunct = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_replaceRepOfPunct['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd50b040f6580ff0df59c6aef4b2bed74fd5a323"},"cell_type":"code","source":"for index, row in text_replaceRepOfPunct.iterrows():\n    row['TextAfter'] = replaceMultiExclamationMark(row['TextBefore'])\n    row['TextAfter'] = replaceMultiQuestionMark(row['TextBefore'])\n    row['TextAfter'] = replaceMultiStopMark(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4d8281d71d95fa7a299593122fdb074863110547"},"cell_type":"code","source":"text_replaceRepOfPunct['Changed'] = np.where(text_replaceRepOfPunct['TextBefore']==text_replaceRepOfPunct['TextAfter'], 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_replaceRepOfPunct[text_replaceRepOfPunct['Changed']=='yes']), len(text_replaceRepOfPunct), 100*len(text_replaceRepOfPunct[text_replaceRepOfPunct['Changed']=='yes'])/len(text_replaceRepOfPunct)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f8305a96b11d645a3ad024b265f00ff4ebdad39"},"cell_type":"code","source":"for index, row in text_replaceRepOfPunct[text_replaceRepOfPunct['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc911c610cc2d7cb845f36c4c1cad7591317c0cc"},"cell_type":"markdown","source":"## 3. Remove Punctuation\n**Example:** Why haven't two democracies never ever went for a full fledged war? What stops them? -> Why havent two democracies never ever went for a full fledged war What stops them"},{"metadata":{"trusted":true,"_uuid":"e8db070a372cc0c04fada1ab95bf01d456abcc24"},"cell_type":"code","source":"import string\ntranslator = str.maketrans('', '', string.punctuation)\ntext_removePunctuation = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_removePunctuation['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70083f413bf6411d7fedb7d7a799dba6efbc03f3"},"cell_type":"code","source":"for index, row in text_removePunctuation.iterrows():\n    row['TextAfter'] = row['TextBefore'].translate(translator) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c96b84ffddd998bc73f641538abab7e498f1ac3"},"cell_type":"code","source":"text_removePunctuation['Changed'] = np.where(text_removePunctuation['TextBefore']==text_removePunctuation['TextAfter'], 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_removePunctuation[text_removePunctuation['Changed']=='yes']), len(text_removePunctuation), 100*len(text_removePunctuation[text_removePunctuation['Changed']=='yes'])/len(text_removePunctuation)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"0fc301c2cc218739bb603f2b22ddaebb483ef54c"},"cell_type":"code","source":"for index, row in text_removePunctuation[text_removePunctuation['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3545ca50d76dbfe5a99c6806b51c60f70979249a"},"cell_type":"markdown","source":"Hmm, i expected everything to change, because they are question with \"?\". Let's see the ones that didn't change."},{"metadata":{"trusted":true,"_uuid":"818a2187bb542210bf93e9467d3ee33319862719"},"cell_type":"code","source":"for index, row in text_removePunctuation[text_removePunctuation['Changed']=='no'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf9fcef515a0e716a6a3c6e6bc44cb4928cd54c1"},"cell_type":"markdown","source":"## 4. Replace Contractions\nThis techniques replaces contractions to their equivalents.\n\n**Example:** What's the scariest thing that ever happened to anyone? -> What is the scariest thing that ever happened to anyone?"},{"metadata":{"trusted":true,"_uuid":"47a8feccbc1364a3c0aa5a35a99e91f233927baa"},"cell_type":"code","source":"contraction_patterns = [ (r'won\\'t', 'will not'), (r'can\\'t', 'cannot'), (r'i\\'m', 'i am'), (r'ain\\'t', 'is not'), (r'(\\w+)\\'ll', '\\g<1> will'), (r'(\\w+)n\\'t', '\\g<1> not'),\n                         (r'(\\w+)\\'ve', '\\g<1> have'), (r'(\\w+)\\'s', '\\g<1> is'), (r'(\\w+)\\'re', '\\g<1> are'), (r'(\\w+)\\'d', '\\g<1> would'), (r'&', 'and'), (r'dammit', 'damn it'), (r'dont', 'do not'), (r'wont', 'will not') ]\ndef replaceContraction(text):\n    patterns = [(re.compile(regex), repl) for (regex, repl) in contraction_patterns]\n    for (pattern, repl) in patterns:\n        (text, count) = re.subn(pattern, repl, text)\n    return text\n\ntext_replaceContractions = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_replaceContractions['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fa9db144bd7fa2dcd793225890a3d90c7ea41175"},"cell_type":"code","source":"for index, row in text_replaceContractions.iterrows():\n    row['TextAfter'] = replaceContraction(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c90b57020aadc67f0911fe31ab6c53da330be78"},"cell_type":"code","source":"text_replaceContractions['Changed'] = np.where(text_replaceContractions['TextBefore']==text_replaceContractions['TextAfter'], 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_replaceContractions[text_replaceContractions['Changed']=='yes']), len(text_replaceContractions), 100*len(text_replaceContractions[text_replaceContractions['Changed']=='yes'])/len(text_replaceContractions)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7b9af939d9e04c2b73d04f645cbfa37e2f48f343"},"cell_type":"code","source":"for index, row in text_replaceContractions[text_replaceContractions['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"475e14c586dd3c18a95a3e2318869bf847033632"},"cell_type":"markdown","source":"## 5. Lowercase\n**Example:** What do you know about Bram Fischer and the Rivonia Trial? -> what do you know about bram fischer and the rivonia trial?"},{"metadata":{"trusted":true,"_uuid":"355e6fc585fd9d49fc37219c14ab14004ffbb6ce"},"cell_type":"code","source":"text_lowercase = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_lowercase['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b39fd5e7afa0a476d08251151a0fdaf3bfe18f5"},"cell_type":"code","source":"for index, row in text_lowercase.iterrows():\n    row['TextAfter'] = row['TextBefore'].lower()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e9926f52d87ece340dee73deeb4d3ceaa11000e"},"cell_type":"code","source":"text_lowercase['Changed'] = np.where(text_lowercase['TextBefore']==text_lowercase['TextAfter'], 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_lowercase[text_lowercase['Changed']=='yes']), len(text_lowercase), 100*len(text_lowercase[text_lowercase['Changed']=='yes'])/len(text_lowercase)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"183b703c8cfe1ea0fa7393158a1fbcebf625f516"},"cell_type":"code","source":"for index, row in text_lowercase[text_lowercase['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ade19cce80271933534a3c1b9089dbc3bb757c34"},"cell_type":"markdown","source":"Some question are written only in lowercase. This happens when they start with a number."},{"metadata":{"trusted":true,"_uuid":"4aa2466287972dc11924194d281e3c856def51a8"},"cell_type":"code","source":"for index, row in text_lowercase[text_lowercase['Changed']=='no'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a0e1e0caefab5a4ef24519dde8bb41d966fa1ffe"},"cell_type":"markdown","source":"## 6. Replace Negations with Antonyms\n**Example:** Why are humans not able to be evolved developing resistance against diseases? -> Why are humans unable to be evolved developing resistance against diseases ?"},{"metadata":{"trusted":true,"_uuid":"af37ca08e1a1ec777c27b1b2c84ce6afb8b2efd4"},"cell_type":"code","source":"import nltk\nfrom nltk.corpus import wordnet\n\ndef replace(word, pos=None):\n    \"\"\" Creates a set of all antonyms for the word and if there is only one antonym, it returns it \"\"\"\n    antonyms = set()\n    for syn in wordnet.synsets(word, pos=pos):\n        for lemma in syn.lemmas():\n            for antonym in lemma.antonyms():\n                antonyms.add(antonym.name())\n    if len(antonyms) == 1:\n        return antonyms.pop()\n    else:\n        return None\n\ndef replaceNegations(text):\n    \"\"\" Finds \"not\" and antonym for the next word and if found, replaces not and the next word with the antonym \"\"\"\n    i, l = 0, len(text)\n    words = []\n    while i < l:\n        word = text[i]\n        if word == 'not' and i+1 < l:\n            ant = replace(text[i+1])\n            if ant:\n                words.append(ant)\n                i += 2\n                continue\n        words.append(word)\n        i += 1\n    return words\n\ndef tokenize1(text):\n    tokens = nltk.word_tokenize(text)\n    tokens = replaceNegations(tokens)\n    text = \" \".join(tokens)\n    return text\n\ntext_replaceNegations = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_replaceNegations['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"479f4b3a4186f024059abed8f26ec8b309df17cf"},"cell_type":"code","source":"for index, row in text_replaceNegations.iterrows():\n    row['TextAfter'] = tokenize1(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"711021a1b0437d1a16e9a634fdd45bc115751bd4"},"cell_type":"code","source":"text_replaceNegations['Changed'] = np.where(text_replaceNegations['TextBefore'].str.replace(\" \",\"\")==text_replaceNegations['TextAfter'].str.replace(\" \",\"\").str.replace(\"``\",'\"').str.replace(\"''\",'\"'), 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_replaceNegations[text_replaceNegations['Changed']=='yes']), len(text_replaceNegations), 100*len(text_replaceNegations[text_replaceNegations['Changed']=='yes'])/len(text_replaceNegations)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"7ece918f2d15be189493537c2bece44ef76b5f83"},"cell_type":"code","source":"for index, row in text_replaceNegations[text_replaceNegations['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4bb1384b130c3a75099dac3fd8700449918c98ca"},"cell_type":"markdown","source":"## 7. Handle Capitalized Words\n**Example:** Which is better to use, Avro or ORC? -> Which is better to use , Avro or ALL_CAPS_ORC ?"},{"metadata":{"trusted":true,"_uuid":"12772ee36dbd66e7ec507959990516590e1f38e6"},"cell_type":"code","source":"def addCapTag(word):\n    \"\"\" Finds a word with at least 3 characters capitalized and adds the tag ALL_CAPS_ \"\"\"\n    if(len(re.findall(\"[A-Z]{3,}\", word))):\n        word = word.replace('\\\\', '' )\n        transformed = re.sub(\"[A-Z]{3,}\", \"ALL_CAPS_\"+word, word)\n        return transformed\n    else:\n        return word\n\ndef tokenize2(text):\n    finalTokens = []\n    tokens = nltk.word_tokenize(text)\n    for w in tokens:\n        finalTokens.append(addCapTag(w))\n    text = \" \".join(finalTokens)\n    return text\n\ntext_handleCapWords = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_handleCapWords['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16e555ff71640676cb33f833fa289f72ec6991db"},"cell_type":"code","source":"for index, row in text_handleCapWords.iterrows():\n    row['TextAfter'] = tokenize2(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a732d04fe19e20103c6bca0d1ccd91245535eeb"},"cell_type":"code","source":"text_handleCapWords['Changed'] = np.where(text_handleCapWords['TextBefore'].str.replace(\" \",\"\")==text_handleCapWords['TextAfter'].str.replace(\" \",\"\").str.replace(\"``\",'\"').str.replace(\"''\",'\"'), 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_handleCapWords[text_handleCapWords['Changed']=='yes']), len(text_handleCapWords), 100*len(text_handleCapWords[text_handleCapWords['Changed']=='yes'])/len(text_handleCapWords)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"82e1c29c084ad1cfb0235b9b294069c9043038c5"},"cell_type":"code","source":"for index, row in text_handleCapWords[text_handleCapWords['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0fd322ed368820947deacbef2b5f9e6dce4042e4"},"cell_type":"markdown","source":"## 8. Remove Stopwords\n**Example:** How I know whether a girl had done sex before sex with me? -> How I know whether girl done sex sex ?"},{"metadata":{"trusted":true,"_uuid":"3a7db86307c657c34c85310aa80b44bc1989c427"},"cell_type":"code","source":"from nltk.corpus import stopwords\nstoplist = stopwords.words('english')\n\ndef tokenize(text):\n    finalTokens = []\n    tokens = nltk.word_tokenize(text)\n    for w in tokens:\n        if (w not in stoplist):\n            finalTokens.append(w)\n    text = \" \".join(finalTokens)\n    return text\n\ntext_removeStopwords = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_removeStopwords['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a6fe3f5a52dff0d3ab120674583a669d3ed712b"},"cell_type":"code","source":"for index, row in text_removeStopwords.iterrows():\n    row['TextAfter'] = tokenize(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e1482892be2e8b5427639d9fc81cf04d07e66ed"},"cell_type":"code","source":"text_removeStopwords['Changed'] = np.where(text_removeStopwords['TextBefore'].str.replace(\" \",\"\")==text_removeStopwords['TextAfter'].str.replace(\" \",\"\").str.replace(\"``\",'\"').str.replace(\"''\",'\"'), 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_removeStopwords[text_removeStopwords['Changed']=='yes']), len(text_removeStopwords), 100*len(text_removeStopwords[text_removeStopwords['Changed']=='yes'])/len(text_removeStopwords)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c88343608802d3734eaf00998a1200992006e607"},"cell_type":"code","source":"for index, row in text_removeStopwords[text_removeStopwords['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25b684706946cec462e5aec250125004eba65dce"},"cell_type":"markdown","source":"## 9. Replace Elongated Words\nThis technique replaces an elongated word with its basic form, unless the word exists in the lexicon.\n\n**Example:** Game of Thrones, what does Arya find out about Littlefinger? -> Game of Thrones , what does Arya find out about Litlefinger ?"},{"metadata":{"trusted":true,"_uuid":"8be40c531d37d808167b2263260774d43479035a"},"cell_type":"code","source":"def replaceElongated(word):\n    \"\"\" Replaces an elongated word with its basic form, unless the word exists in the lexicon \"\"\"\n\n    repeat_regexp = re.compile(r'(\\w*)(\\w)\\2(\\w*)')\n    repl = r'\\1\\2\\3'\n    if wordnet.synsets(word):\n        return word\n    repl_word = repeat_regexp.sub(repl, word)\n    if repl_word != word:      \n        return replaceElongated(repl_word)\n    else:       \n        return repl_word\n    \ndef tokenize(text):\n    finalTokens = []\n    tokens = nltk.word_tokenize(text)\n    for w in tokens:\n        finalTokens.append(replaceElongated(w))\n    text = \" \".join(finalTokens)\n    return text\n\ntext_removeElWords = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_removeElWords['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e6ed9fe42bbb3db2fd908fd75f3f54a68d94176"},"cell_type":"code","source":"for index, row in text_removeElWords.iterrows():\n    row['TextAfter'] = tokenize(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b815a11ab55e4533f0e3c0642630c35c309feb72"},"cell_type":"code","source":"text_removeElWords['Changed'] = np.where(text_removeElWords['TextBefore'].str.replace(\" \",\"\")==text_removeElWords['TextAfter'].str.replace(\" \",\"\").str.replace(\"``\",'\"').str.replace(\"''\",'\"'), 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_removeElWords[text_removeElWords['Changed']=='yes']), len(text_removeElWords), 100*len(text_removeElWords[text_removeElWords['Changed']=='yes'])/len(text_removeElWords)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dcc2d2413e8b98ca26bbff3fd84c677a427ffecf"},"cell_type":"code","source":"for index, row in text_removeElWords[text_removeElWords['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c478fb8e3810e1aa6385ae17cf026027e4d5f06"},"cell_type":"markdown","source":"## 10. Stemming/Lemmatizing\n**Example:** How do modern military submarines reduce noise to achieve stealth? -> how do modern militari submarin reduc nois to achiev stealth ?"},{"metadata":{"trusted":true,"_uuid":"432ef1f008675e538b49d9c4aec479647638c093"},"cell_type":"code","source":"from nltk.stem.porter import PorterStemmer\nstemmer = PorterStemmer() #set stemmer\nfrom nltk.stem import WordNetLemmatizer\nlemmatizer = WordNetLemmatizer() # set lemmatizer\n\ndef tokenize(text):\n    finalTokens = []\n    tokens = nltk.word_tokenize(text)\n    for w in tokens:\n        finalTokens.append(stemmer.stem(w)) # change this to lemmatizer.lemmatize(w) for Lemmatizing\n    text = \" \".join(finalTokens)\n    return text\n\ntext_stemming = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_stemming['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c9b54b331016c0c7f41df1fd21181fb66a09121"},"cell_type":"code","source":"for index, row in text_stemming.iterrows():\n    row['TextAfter'] = tokenize(row['TextBefore'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8fa2ee8a86cc0a92088afca8a679ff0e63d461cb"},"cell_type":"code","source":"text_stemming['Changed'] = np.where(text_stemming['TextBefore'].str.replace(\" \",\"\")==text_stemming['TextAfter'].str.replace(\" \",\"\").str.replace(\"``\",'\"').str.replace(\"''\",'\"'), 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_stemming[text_stemming['Changed']=='yes']), len(text_stemming), 100*len(text_stemming[text_stemming['Changed']=='yes'])/len(text_stemming)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f602787bee9ab08e45590878f4f208c6d9092a25"},"cell_type":"code","source":"for index, row in text_stemming[text_stemming['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38c7bd7939a3295d50551d229a4aa8ac7d48f42b"},"cell_type":"markdown","source":"## Combos\nOf course we can use more than one technique at the same time. The order is essential here.\n\n**Example:** What are the recommended 2D game engines for a beginning Python programmer? -> what recommend d game engin begin python programm"},{"metadata":{"trusted":true,"_uuid":"9fa801a5bfdb1a57e96ac606474f50c74f840e70"},"cell_type":"code","source":"def tokenize(text):\n    finalTokens = []\n    tokens = nltk.word_tokenize(text)\n    for w in tokens:\n        if (w not in stoplist):\n            w = addCapTag(w) # Handle Capitalized Words\n            w = w.lower() # Lowercase\n            w = replaceElongated(w) # Replace Elongated Words\n            w = stemmer.stem(w) # Stemming\n            finalTokens.append(w)\n    text = \" \".join(finalTokens)\n    return text\n\ntext_combos = pd.DataFrame(columns=['TextBefore', 'TextAfter', 'Changed'])\ntext_combos['TextBefore'] = text.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aac4438da5b434c25db3c0c5c7208b63ae4a0ad8"},"cell_type":"code","source":"for index, row in text_combos.iterrows():\n    row['TextAfter'] = replaceContraction(row['TextBefore']) # Replace Contractions\n    row['TextAfter'] = removeNumbers(row['TextAfter']) # Remove Integers\n    row['TextAfter'] = replaceMultiExclamationMark(row['TextAfter']) # Replace Multi Exclamation Marks\n    row['TextAfter'] = replaceMultiQuestionMark(row['TextAfter']) # Replace Multi Question Marks\n    row['TextAfter'] = replaceMultiStopMark(row['TextAfter']) # Repalce Multi Stop Marks\n    row['TextAfter'] = row['TextAfter'].translate(translator) # Remove Punctuation\n    row['TextAfter'] = tokenize(row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4d5cb507785c24065408f8dd17202a7accb24ca"},"cell_type":"code","source":"text_combos['Changed'] = np.where(text_combos['TextBefore'].str.replace(\" \",\"\")==text_combos['TextAfter'].str.replace(\" \",\"\").str.replace(\"``\",'\"').str.replace(\"''\",'\"'), 'no', 'yes')\nprint(\"{} of {} ({:.4f}%) questions have been changed.\".format(len(text_combos[text_combos['Changed']=='yes']), len(text_combos), 100*len(text_combos[text_combos['Changed']=='yes'])/len(text_combos)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6607da645a7152b09a2dafe377351d8185326aa5"},"cell_type":"code","source":"for index, row in text_combos[text_combos['Changed']=='yes'].head().iterrows():\n    print(row['TextBefore'],'->',row['TextAfter'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7df41aecf5dd758bfd0b80f08fa0a2533df66c29"},"cell_type":"markdown","source":"Thank you for reaching this point! Hope you enjoyed it! Your upvote will be much appreciated!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}