{"cells":[{"metadata":{"_uuid":"4e904b26a9929bdb57d51c124c8065dd14414858"},"cell_type":"markdown","source":"In this notebook, I am going to sharing some text cleaning mostly based on bad case analyse. The cleaning methods include:\n\n- remove empty space\n- replace strange punctuations and raplace diacritics\n- clean numbers, split numbers and letters\n- de-contract the contraction\n- clean math latex\n- clean many misspelling words\n- spacing punctuations expect `-`and `.`\n- **fix `-`and `.` punc spacing bugs which common kernels existed**\n- spacing some connected words like `whatis, whichcountry`\n- clean repeated letters in one word, like `hoooooow`\n- clean some masked words like `f***, st*up*id`\n\nReference:\n- [How to: Preprocessing when using embeddings](https://www.kaggle.com/christofhenkel/how-to-preprocessing-when-using-embeddings)\n- [Improve your Score with some Text Preprocessing](https://www.kaggle.com/theoviel/improve-your-score-with-some-text-preprocessing)\n\nCleaning text data has caused me huge time! **Please UPVOTE if you find it useful**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport string\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport operator\nfrom gensim.models import KeyedVectors\nimport matplotlib.pyplot as plt\nimport seaborn as sns\ncolor = sns.color_palette()\n%matplotlib inline\nfrom tqdm import tqdm\ntqdm.pandas()\nimport pickle\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import psutil\nfrom multiprocessing import Pool\n\nnum_partitions = 20  # number of partitions to split dataframe\nnum_cores = psutil.cpu_count()  # number of cores on your machine\n\nprint('number of cores:', num_cores)\ndef df_parallelize_run(df, func):\n    df_split = np.array_split(df, num_partitions)\n    pool = Pool(num_cores)\n    df = pd.concat(pool.map(func, df_split))\n    pool.close()\n    pool.join()\n    return df\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9a525b31fed40399095c0538f5aac361e97be86a"},"cell_type":"markdown","source":"# Load dataset and Embeddings"},{"metadata":{"trusted":true,"_uuid":"f806e0c1329075669aae1d2a3391f7aca99121b0"},"cell_type":"code","source":"%%time\ntrain_df = pd.read_csv(\"../input/train.csv\", encoding='utf8')\ntest_df = pd.read_csv(\"../input/test.csv\", encoding='utf8')\nall_test_texts = ''.join(test_df.question_text.values.tolist())\n\nprint('Train:', train_df.shape)\nprint('Test:', test_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e0ac21c0fa978a73eae298ec9bb8a29022bc90f"},"cell_type":"code","source":"train_ques_lens = train_df['question_text'].map(lambda x: len(x.split(' ')))\ntest_ques_lens = test_df['question_text'].map(lambda x: len(x.split(' ')))\ntrain_len_sts = train_ques_lens.describe().reset_index().rename(columns={'index':'train_stat'})\ntrain_len_sts['question_text'] = train_len_sts['question_text'].astype(int)\ntest_len_sts = test_ques_lens.describe().reset_index().rename(columns={'index':'test_stat'})\ntest_len_sts['question_text'] = test_len_sts['question_text'].astype(int)\n\nlen_sts = pd.concat([train_len_sts, test_len_sts], axis=1)\ndisplay(len_sts)\n\ndel train_ques_lens; del test_ques_lens; del train_len_sts; del test_len_sts\ngc.collect()\npass","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d194d2150e4879f0824a2651f120b6a6b0b27cd"},"cell_type":"code","source":"def load_embed(file):\n    def get_coefs(word,*arr): \n        return word, np.asarray(arr, dtype='float16')\n    \n    if file == '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec':\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file) if len(o)>100)\n    elif file == '../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin':\n        embeddings_index = KeyedVectors.load_word2vec_format(file, binary=True)\n    else:\n        embeddings_index = dict(get_coefs(*o.split(\" \")) for o in open(file, encoding='latin'))\n        \n    return embeddings_index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a52952bf06bc4076484ec0152f2323145c38b26c"},"cell_type":"code","source":"%%time\nglove = '../input/embeddings/glove.840B.300d/glove.840B.300d.txt'\nparagram =  '../input/embeddings/paragram_300_sl999/paragram_300_sl999.txt'\nwiki_news = '../input/embeddings/wiki-news-300d-1M/wiki-news-300d-1M.vec'\ngoogle_path = '../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin'\n\nprint(\"Extracting GloVe embedding\")\nembed_glove = load_embed(glove)\nprint(\"Extracting Paragram embedding\")\nembed_paragram = load_embed(paragram)\nprint(\"Extracting FastText embedding\")\nembed_fasttext = load_embed(wiki_news)\nprint(\"Extracting GoogleNews embedding\")\nembed_google = load_embed(google_path)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d096abeda0515165caa62aa098cd0dbd55f5292"},"cell_type":"markdown","source":"# Word coverage function"},{"metadata":{"trusted":true,"_uuid":"c091aa536356ad82d47e1919ecbf19b451a6b5cf"},"cell_type":"code","source":"def build_vocab(texts):\n    sentences = texts.apply(lambda x: x.split()).values\n    vocab = {}\n    for sentence in sentences:\n        for word in sentence:\n            try:\n                vocab[word] += 1\n            except KeyError:\n                vocab[word] = 1\n    return vocab\n\ndef check_coverage(vocab, embeddings_index):\n    known_words = {}\n    unknown_words = {}\n    nb_known_words = 0\n    nb_unknown_words = 0\n    for word in vocab.keys():\n        if word in embeddings_index:\n            known_words[word] = embeddings_index[word]\n            nb_known_words += vocab[word]\n        elif word.capitalize() in embeddings_index:\n            known_words[word] = embeddings_index[word.capitalize()]\n            nb_known_words += vocab[word]\n        elif word.lower() in embeddings_index:\n            known_words[word] = embeddings_index[word.lower()]\n            nb_known_words += vocab[word]\n        elif word.upper() in embeddings_index:\n            known_words[word] = embeddings_index[word.upper()]\n            nb_known_words += vocab[word]\n        else:\n            unknown_words[word] = vocab[word]\n            nb_unknown_words += vocab[word]\n\n    print('Found embeddings for {:.2%} of vocab'.format(len(known_words) / len(vocab)))\n    print('Found embeddings for  {:.2%} of all text'.format(nb_known_words / (nb_known_words + nb_unknown_words)))\n    unknown_words = sorted(unknown_words.items(), key=operator.itemgetter(1))[::-1]\n\n    return unknown_words\n\ndef vocab_check_coverage(train_df, test_df):\n    df = pd.concat([train_df.drop('target', axis=1), test_df]).reset_index(drop=True)\n    \n    vocab = build_vocab(df['question_text'])\n    print(\"Glove : \")\n    oov_glove = check_coverage(vocab, embed_glove)\n    oov_glove = {\"oov_rate\": len(oov_glove) / len(vocab), 'oov_words': oov_glove}\n    print(\"Paragram : \")\n    oov_paragram = check_coverage(vocab, embed_paragram)\n    oov_paragram = {\"oov_rate\": len(oov_paragram) / len(vocab), 'oov_words': oov_paragram}\n    print(\"FastText : \")\n    oov_fasttext = check_coverage(vocab, embed_fasttext)\n    oov_fasttext = {\"oov_rate\": len(oov_fasttext) / len(vocab), 'oov_words': oov_fasttext}\n    print(\"Google : \")\n    oov_google = check_coverage(vocab, embed_google)\n    oov_google = {\"oov_rate\": len(oov_google) / len(vocab), 'oov_words': oov_google}\n    \n    return oov_glove, oov_paragram, oov_fasttext, oov_google\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc72abdac6dfe5fe33bcd8795e90230bd79341dc"},"cell_type":"markdown","source":"# Original word coverage"},{"metadata":{"trusted":true,"_uuid":"5a427d1e1bafa963eef8e2097eef6131cba4dabb"},"cell_type":"code","source":"%%time\noov_glove, oov_paragram, oov_fasttext, oov_google = vocab_check_coverage(train_df, test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8862f7ccbc7374788fadff1dd4b47a6fc914efc7"},"cell_type":"code","source":"print('glove oov rate:', oov_glove['oov_rate'])\nprint('paragram oov rate:', oov_paragram['oov_rate'])\nprint('fasttext oov rate:', oov_fasttext['oov_rate'])\nprint('google oov rate:', oov_google['oov_rate'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b47d2420d79368186b8fdb2cee58bc664c83a447"},"cell_type":"markdown","source":"# Text Cleaning"},{"metadata":{"trusted":true,"_uuid":"44ec1a11ec2ed9916f32869ea3491aa3edfad710"},"cell_type":"code","source":"# remove space\nspaces = ['\\u200b', '\\u200e', '\\u202a', '\\u202c', '\\ufeff', '\\uf0d8', '\\u2061', '\\x10', '\\x7f', '\\x9d', '\\xad', '\\xa0']\ndef remove_space(text):\n    \"\"\"\n    remove extra spaces and ending space if any\n    \"\"\"\n    for space in spaces:\n        text = text.replace(space, ' ')\n    text = text.strip()\n    text = re.sub('\\s+', ' ', text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"478b59102e0ff30263c847f2ca9875abd50f98a2"},"cell_type":"code","source":"# replace strange punctuations and raplace diacritics\nfrom unicodedata import category, name, normalize\n\ndef remove_diacritics(s):\n    return ''.join(c for c in normalize('NFKD', s.replace('ø', 'o').replace('Ø', 'O').replace('⁻', '-').replace('₋', '-'))\n                  if category(c) != 'Mn')\n\nspecial_punc_mappings = {\"—\": \"-\", \"–\": \"-\", \"_\": \"-\", '”': '\"', \"″\": '\"', '“': '\"', '•': '.', '−': '-',\n                         \"’\": \"'\", \"‘\": \"'\", \"´\": \"'\", \"`\": \"'\", '\\u200b': ' ', '\\xa0': ' ','،':'','„':'',\n                         '…': ' ... ', '\\ufeff': ''}\ndef clean_special_punctuations(text):\n    for punc in special_punc_mappings:\n        if punc in text:\n            text = text.replace(punc, special_punc_mappings[punc])\n    # 注意顺序，remove_diacritics放前面会导致 'don´t' 被处理为 'don t'\n    text = remove_diacritics(text)\n    return text\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5c9c9400a176a6ae854566346fe6a4a9c17927ce"},"cell_type":"code","source":"# clean numbers\ndef clean_number(text):\n    # 字母和数字分开\n    text = re.sub(r'(\\d+)([a-zA-Z])', '\\g<1> \\g<2>', text)\n    text = re.sub(r'(\\d+) (th|st|nd|rd) ', '\\g<1>\\g<2> ', text)\n    text = re.sub(r'(\\d+),(\\d+)', '\\g<1>\\g<2>', text)\n    \n#     text = re.sub('[0-9]{5,}', '#####', text)\n#     text = re.sub('[0-9]{4}', '####', text)\n#     text = re.sub('[0-9]{3}', '###', text)\n#     text = re.sub('[0-9]{2}', '##', text)\n    \n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ac8d04187babcfb99e68cbb4b78c3a837e16a00"},"cell_type":"code","source":"# 进行 decontracted 之前，清洗不常见的缩略词，如 U.S.\nrare_words_mapping = {' s.p ': ' ', ' S.P ': ' ', 'U.s.p': '', 'U.S.A.': 'USA', 'u.s.a.': 'USA', 'U.S.A': 'USA',\n                      'u.s.a': 'USA', 'U.S.': 'USA', 'u.s.': 'USA', ' U.S ': ' USA ', ' u.s ': ' USA ', 'U.s.': 'USA',\n                      ' U.s ': 'USA', ' u.S ': ' USA ', 'fu.k': 'fuck', 'U.K.': 'UK', ' u.k ': ' UK ',\n                      ' don t ': ' do not ', 'bacteries': 'batteries', ' yr old ': ' years old ', 'Ph.D': 'PhD',\n                      'cau.sing': 'causing', 'Kim Jong-Un': 'The president of North Korea', 'savegely': 'savagely',\n                      'Ra apist': 'Rapist', '2fifth': 'twenty fifth', '2third': 'twenty third',\n                      '2nineth': 'twenty nineth', '2fourth': 'twenty fourth', '#metoo': 'MeToo',\n                      'Trumpcare': 'Trump health care system', '4fifth': 'forty fifth', 'Remainers': 'remainder',\n                      'Terroristan': 'terrorist', 'antibrahmin': 'anti brahmin',\n                      'fuckboys': 'fuckboy', 'Fuckboys': 'fuckboy', 'Fuckboy': 'fuckboy', 'fuckgirls': 'fuck girls',\n                      'fuckgirl': 'fuck girl', 'Trumpsters': 'Trump supporters', '4sixth': 'forty sixth',\n                      'culturr': 'culture',\n                      'weatern': 'western', '4fourth': 'forty fourth', 'emiratis': 'emirates', 'trumpers': 'Trumpster',\n                      'indans': 'indians', 'mastuburate': 'masturbate', 'f**k': 'fuck', 'F**k': 'fuck', 'F**K': 'fuck',\n                      ' u r ': ' you are ', ' u ': ' you ', '操你妈': 'fuck your mother', 'e.g.': 'for example',\n                      'i.e.': 'in other words', '...': '.', 'et.al': 'elsewhere', 'anti-Semitic': 'anti-semitic',\n                      'f***': 'fuck', 'f**': 'fuc', 'F***': 'fuck', 'F**': 'fuc',\n                      'a****': 'assho', 'a**': 'ass', 'h***': 'hole', 'A****': 'assho', 'A**': 'ass', 'H***': 'hole',\n                      's***': 'shit', 's**': 'shi', 'S***': 'shit', 'S**': 'shi', 'Sh**': 'shit',\n                      'p****': 'pussy', 'p*ssy': 'pussy', 'P****': 'pussy',\n                      'p***': 'porn', 'p*rn': 'porn', 'P***': 'porn',\n                      'st*up*id': 'stupid',\n                      'd***': 'dick', 'di**': 'dick', 'h*ck': 'hack',\n                      'b*tch': 'bitch', 'bi*ch': 'bitch', 'bit*h': 'bitch', 'bitc*': 'bitch', 'b****': 'bitch',\n                      'b***': 'bitc', 'b**': 'bit', 'b*ll': 'bull'\n                      }\n\n\ndef pre_clean_rare_words(text):\n    for rare_word in rare_words_mapping:\n        if rare_word in text:\n            text = text.replace(rare_word, rare_words_mapping[rare_word])\n\n    return text\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3fafaf952b73736ae8e3f1ecbab8b253c404dad1"},"cell_type":"code","source":"# de-contract the contraction\ndef decontracted(text):\n    # specific\n    text = re.sub(r\"(W|w)on(\\'|\\’)t \", \"will not \", text)\n    text = re.sub(r\"(C|c)an(\\'|\\’)t \", \"can not \", text)\n    text = re.sub(r\"(Y|y)(\\'|\\’)all \", \"you all \", text)\n    text = re.sub(r\"(Y|y)a(\\'|\\’)ll \", \"you all \", text)\n\n    # general\n    text = re.sub(r\"(I|i)(\\'|\\’)m \", \"i am \", text)\n    text = re.sub(r\"(A|a)in(\\'|\\’)t \", \"is not \", text)\n    text = re.sub(r\"n(\\'|\\’)t \", \" not \", text)\n    text = re.sub(r\"(\\'|\\’)re \", \" are \", text)\n    text = re.sub(r\"(\\'|\\’)s \", \" is \", text)\n    text = re.sub(r\"(\\'|\\’)d \", \" would \", text)\n    text = re.sub(r\"(\\'|\\’)ll \", \" will \", text)\n    text = re.sub(r\"(\\'|\\’)t \", \" not \", text)\n    text = re.sub(r\"(\\'|\\’)ve \", \" have \", text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2388df4518faa334f36076767f89a494ee6dea8a"},"cell_type":"code","source":"def clean_latex(text):\n    \"\"\"\n    convert r\"[math]\\vec{x} + \\vec{y}\" to English\n    \"\"\"\n    # edge case\n    text = re.sub(r'\\[math\\]', ' LaTex math ', text)\n    text = re.sub(r'\\[\\/math\\]', ' LaTex math ', text)\n    text = re.sub(r'\\\\', ' LaTex ', text)\n\n    pattern_to_sub = {\n        r'\\\\mathrm': ' LaTex math mode ',\n        r'\\\\mathbb': ' LaTex math mode ',\n        r'\\\\boxed': ' LaTex equation ',\n        r'\\\\begin': ' LaTex equation ',\n        r'\\\\end': ' LaTex equation ',\n        r'\\\\left': ' LaTex equation ',\n        r'\\\\right': ' LaTex equation ',\n        r'\\\\(over|under)brace': ' LaTex equation ',\n        r'\\\\text': ' LaTex equation ',\n        r'\\\\vec': ' vector ',\n        r'\\\\var': ' variable ',\n        r'\\\\theta': ' theta ',\n        r'\\\\mu': ' average ',\n        r'\\\\min': ' minimum ',\n        r'\\\\max': ' maximum ',\n        r'\\\\sum': ' + ',\n        r'\\\\times': ' * ',\n        r'\\\\cdot': ' * ',\n        r'\\\\hat': ' ^ ',\n        r'\\\\frac': ' / ',\n        r'\\\\div': ' / ',\n        r'\\\\sin': ' Sine ',\n        r'\\\\cos': ' Cosine ',\n        r'\\\\tan': ' Tangent ',\n        r'\\\\infty': ' infinity ',\n        r'\\\\int': ' integer ',\n        r'\\\\in': ' in ',\n    }\n    # post process for look up\n    pattern_dict = {k.strip('\\\\'): v for k, v in pattern_to_sub.items()}\n    # init re\n    patterns = pattern_to_sub.keys()\n    pattern_re = re.compile('(%s)' % '|'.join(patterns))\n\n    def _replace(match):\n        \"\"\"\n        reference: https://www.kaggle.com/hengzheng/attention-capsule-why-not-both-lb-0-694 # noqa\n        \"\"\"\n        try:\n            word = pattern_dict.get(match.group(0).strip('\\\\'))\n        except KeyError:\n            word = match.group(0)\n            print('!!Error: Could Not Find Key: {}'.format(word))\n        return word\n    return pattern_re.sub(_replace, text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3851cac8bb9736b6b1fbdd16b280b26e9f353db2"},"cell_type":"code","source":"# clean misspelling words\nmisspell_mapping = {'Terroristan': 'terrorist Pakistan', 'terroristan': 'terrorist Pakistan',\n                    'FATF': 'Western summit conference',\n                    'BIMARU': 'BIMARU Bihar, Madhya Pradesh, Rajasthan, Uttar Pradesh', 'Hinduphobic': 'Hindu phobic',\n                    'hinduphobic': 'Hindu phobic', 'Hinduphobia': 'Hindu phobic', 'hinduphobia': 'Hindu phobic',\n                    'Babchenko': 'Arkady Arkadyevich Babchenko faked death', 'Boshniaks': 'Bosniaks',\n                    'Dravidanadu': 'Dravida Nadu', 'mysoginists': 'misogynists', 'MGTOWS': 'Men Going Their Own Way',\n                    'mongloid': 'Mongoloid', 'unsincere': 'insincere', 'meninism': 'male feminism',\n                    'jewplicate': 'jewish replicate', 'jewplicates': 'jewish replicate', 'andhbhakts': 'and Bhakt',\n                    'unoin': 'Union', 'daesh': 'Islamic State of Iraq and the Levant', 'burnol': 'movement about Modi',\n                    'Kalergi': 'Coudenhove-Kalergi', 'Bhakts': 'Bhakt', 'bhakts': 'Bhakt', 'Tambrahms': 'Tamil Brahmin',\n                    'Pahul': 'Amrit Sanskar', 'SJW': 'social justice warrior', 'SJWs': 'social justice warrior',\n                    ' incel': ' involuntary celibates', ' incels': ' involuntary celibates', 'emiratis': 'Emiratis',\n                    'weatern': 'western', 'westernise': 'westernize', 'Pizzagate': 'debunked conspiracy theory',\n                    'naïve': 'naive', 'Skripal': 'Russian military officer', 'Skripals': 'Russian military officer',\n                    'Remainers': 'British remainer', 'Novichok': 'Soviet Union agents',\n                    'gauri lankesh': 'Famous Indian Journalist', 'Castroists': 'Castro supporters',\n                    'remainers': 'British remainer', 'bremainer': 'British remainer', 'antibrahmin': 'anti Brahminism',\n                    'HYPSM': ' Harvard, Yale, Princeton, Stanford, MIT', 'HYPS': ' Harvard, Yale, Princeton, Stanford',\n                    'kompromat': 'compromising material', 'Tharki': 'pervert', 'tharki': 'pervert',\n                    'mastuburate': 'masturbate', 'Zoë': 'Zoe', 'indans': 'Indian', ' xender': ' gender',\n                    'Naxali ': 'Naxalite ', 'Naxalities': 'Naxalites', 'Bathla': 'Namit Bathla',\n                    'Mewani': 'Indian politician Jignesh Mevani', 'Wjy': 'Why',\n                    'Fadnavis': 'Indian politician Devendra Fadnavis', 'Awadesh': 'Indian engineer Awdhesh Singh',\n                    'Awdhesh': 'Indian engineer Awdhesh Singh', 'Khalistanis': 'Sikh separatist movement',\n                    'madheshi': 'Madheshi', 'BNBR': 'Be Nice, Be Respectful',\n                    'Jair Bolsonaro': 'Brazilian President politician', 'XXXTentacion': 'Tentacion',\n                    'Slavoj Zizek': 'Slovenian philosopher',\n                    'borderliners': 'borderlines', 'Brexit': 'British Exit', 'Brexiter': 'British Exit supporter',\n                    'Brexiters': 'British Exit supporters', 'Brexiteer': 'British Exit supporter',\n                    'Brexiteers': 'British Exit supporters', 'Brexiting': 'British Exit',\n                    'Brexitosis': 'British Exit disorder', 'brexit': 'British Exit',\n                    'brexiters': 'British Exit supporters', 'jallikattu': 'Jallikattu', 'fortnite': 'Fortnite',\n                    'Swachh': 'Swachh Bharat mission campaign ', 'Quorans': 'Quora users', 'Qoura': 'Quora',\n                    'quoras': 'Quora', 'Quroa': 'Quora', 'QUORA': 'Quora', 'Stupead': 'stupid',\n                    'narcissit': 'narcissist', 'trigger nometry': 'trigonometry',\n                    'trigglypuff': 'student Criticism of Conservatives', 'peoplelook': 'people look',\n                    'paedophelia': 'paedophilia', 'Uogi': 'Yogi', 'adityanath': 'Adityanath',\n                    'Yogi Adityanath': 'Indian monk and Hindu nationalist politician',\n                    'Awdhesh Singh': 'Commissioner of India', 'Doklam': 'Tibet', 'Drumpf ': 'Donald Trump fool ',\n                    'Drumpfs': 'Donald Trump fools', 'Strzok': 'Hillary Clinton scandal', 'rohingya': 'Rohingya ',\n                    ' wumao ': ' cheap Chinese stuff ', 'wumaos': 'cheap Chinese stuff', 'Sanghis': 'Sanghi',\n                    'Tamilans': 'Tamils', 'biharis': 'Biharis', 'Rejuvalex': 'hair growth formula Medicine',\n                    'Fekuchand': 'PM Narendra Modi in India', 'feku': 'Feku', 'Chaiwala': 'tea seller in India',\n                    'Feku': 'PM Narendra Modi in India ', 'deplorables': 'deplorable', 'muhajirs': 'Muslim immigrant',\n                    'Gujratis': 'Gujarati', 'Chutiya': 'Tibet people ', 'Chutiyas': 'Tibet people ',\n                    'thighing': 'masterbate between the legs of a female infant', '卐': 'Nazi Germany',\n                    'Pribumi': 'Native Indonesian', 'Gurmehar': 'Gurmehar Kaur Indian student activist',\n                    'Khazari': 'Khazars', 'Demonetization': 'demonetization', 'demonetisation': 'demonetization',\n                    'demonitisation': 'demonetization', 'demonitization': 'demonetization',\n                    'antinationals': 'antinational', 'Cryptocurrencies': 'cryptocurrency',\n                    'cryptocurrencies': 'cryptocurrency', 'Hindians': 'North Indian', 'Hindian': 'North Indian',\n                    'vaxxer': 'vocal nationalist ', 'remoaner': 'remainer ', 'bremoaner': 'British remainer ',\n                    'Jewism': 'Judaism', 'Eroupian': 'European', \"J & K Dy CM H ' ble Kavinderji\": '',\n                    'WMAF': 'White male married Asian female', 'AMWF': 'Asian male married White female',\n                    'moeslim': 'Muslim', 'cishet': 'cisgender and heterosexual person', 'Eurocentrics': 'Eurocentrism',\n                    'Eurocentric': 'Eurocentrism', 'Afrocentrics': 'Africa centrism', 'Afrocentric': 'Africa centrism',\n                    'Jewdar': 'Jew dar', 'marathis': 'Marathi', 'Gynophobic': 'Gyno phobic',\n                    'Trumpanzees': 'Trump chimpanzee fool', 'Crimean': 'Crimea people ', 'atrracted': 'attract',\n                    'Myeshia': 'widow of Green Beret killed in Niger', 'demcoratic': 'Democratic', 'raaping': 'raping',\n                    'feminazism': 'feminism nazi', 'langague': 'language', 'sathyaraj': 'actor',\n                    'Hongkongese': 'HongKong people', 'hongkongese': 'HongKong people', 'Kashmirians': 'Kashmirian',\n                    'Chodu': 'fucker', 'penish': 'penis',\n                    'chitpavan konkanastha': 'Hindu Maharashtrian Brahmin community',\n                    'Madridiots': 'Real Madrid idiot supporters', 'Ambedkarite': 'Dalit Buddhist movement ',\n                    'ReleaseTheMemo': 'cry for the right and Trump supporters', 'harrase': 'harass',\n                    'Barracoon': 'Black slave', 'Castrater': 'castration', 'castrater': 'castration',\n                    'Rapistan': 'Pakistan rapist', 'rapistan': 'Pakistan rapist', 'Turkified': 'Turkification',\n                    'turkified': 'Turkification', 'Dumbassistan': 'dumb ass Pakistan', 'facetards': 'Facebook retards',\n                    'rapefugees': 'rapist refugee', 'Khortha': 'language in the Indian state of Jharkhand',\n                    'Magahi': 'language in the northeastern Indian', 'Bajjika': 'language spoken in eastern India',\n                    'superficious': 'superficial', 'Sense8': 'American science fiction drama web television series',\n                    'Saipul Jamil': 'Indonesia artist', 'bhakht': 'bhakti', 'Smartia': 'dumb nation',\n                    'absorve': 'absolve', 'citicise': 'criticize', 'youtu ': 'youtube ', 'Whta': 'What',\n                    'esspecial': 'especial', 'doI': 'do I', 'theBest': 'the best',\n                    'howdoes': 'how does', 'Etherium': 'Ethereum', '2k17': '2017', '2k18': '2018', 'qiblas': 'qibla',\n                    'Hello4 2 cab': 'Online Cab Booking', 'bodyshame': 'body shaming', 'bodyshoppers': 'body shopping',\n                    'bodycams': 'body cams', 'Cananybody': 'Can any body', 'deadbody': 'dead body',\n                    'deaddict': 'de addict', 'Northindian': 'North Indian ', 'northindian': 'north Indian ',\n                    'northkorea': 'North Korea', 'koreaboo': 'Korea boo ',\n                    'Brexshit': 'British Exit bullshit', 'shitpost': 'shit post', 'shitslam': 'shit Islam',\n                    'shitlords': 'shit lords', 'Fck': 'Fuck', 'Clickbait': 'click bait ', 'clickbait': 'click bait ',\n                    'mailbait': 'mail bait', 'healhtcare': 'healthcare', 'trollbots': 'troll bots',\n                    'trollled': 'trolled', 'trollimg': 'trolling', 'cybertrolling': 'cyber trolling',\n                    'sickular': 'India sick secular ', 'Idiotism': 'idiotism',\n                    'Niggerism': 'Nigger', 'Niggeriah': 'Nigger'}\n\ndef clean_misspell(text):\n    for bad_word in misspell_mapping:\n        if bad_word in text:\n            text = text.replace(bad_word, misspell_mapping[bad_word])\n    return text\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1ae23251cc330a3f636ee6c342b259d23b824b1a"},"cell_type":"markdown","source":"All of the cleaning kernels I found that, they add space around all punctuations. There is a litter bug when spacing dash `-` and point `.`, it will drop the word coverage. For example, the words like `self-trust`, `Michelin-starred` and `pro-Islamic` these words already have pretrained embeddings, force break these words will decrease the word coverage! **I have found 18128 words like this.**"},{"metadata":{"trusted":true,"_uuid":"3e397c5dc41ff48578b9c520e5a97156ed64fd08"},"cell_type":"code","source":"regular_punct = list(string.punctuation)\nextra_punct = [\n    ',', '.', '\"', ':', ')', '(', '!', '?', '|', ';', \"'\", '$', '&',\n    '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£',\n    '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',\n    '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', '“', '★', '”',\n    '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾',\n    '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', '▒', '：', '¼', '⊕', '▼',\n    '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲',\n    'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', '∙', '）', '↓', '、', '│', '（', '»',\n    '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø',\n    '¹', '≤', '‡', '√', '«', '»', '´', 'º', '¾', '¡', '§', '£', '₤']\nall_punct = list(set(regular_punct + extra_punct))\n# do not spacing - and .\nall_punct.remove('-')\nall_punct.remove('.')\n\ndef spacing_punctuation(text):\n    \"\"\"\n    add space before and after punctuation and symbols\n    \"\"\"\n    for punc in all_punct:\n        if punc in text:\n            text = text.replace(punc, f' {punc} ')\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cbaa748e5a6348aa59b2825f0f5e1d4360a235f9"},"cell_type":"code","source":"# spell check and according to bad case analyse\nbad_case_words = {'jewprofits': 'jew profits', 'QMAS': 'Quality Migrant Admission Scheme', 'casterating': 'castrating',\n                  'Kashmiristan': 'Kashmir', 'CareOnGo': 'India first and largest Online distributor of medicines',\n                  'Setya Novanto': 'a former Indonesian politician', 'TestoUltra': 'male sexual enhancement supplement',\n                  'rammayana': 'ramayana', 'Badaganadu': 'Brahmin community that mainly reside in Karnataka',\n                  'bitcjes': 'bitches', 'mastubrate': 'masturbate', 'Français': 'France',\n                  'Adsresses': 'address', 'flemmings': 'flemming', 'intermate': 'inter mating', 'feminisam': 'feminism',\n                  'cuckholdry': 'cuckold', 'Niggor': 'black hip-hop and electronic artist', 'narcsissist': 'narcissist',\n                  'Genderfluid': 'Gender fluid', ' Im ': ' I am ', ' dont ': ' do not ', 'Qoura': 'Quora',\n                  'ethethnicitesnicites': 'ethnicity', 'Namit Bathla': 'Content Writer', 'What sApp': 'WhatsApp',\n                  'Führer': 'Fuhrer', 'covfefe': 'coverage', 'accedentitly': 'accidentally', 'Cuckerberg': 'Zuckerberg',\n                  'transtrenders': 'incredibly disrespectful to real transgender people',\n                  'frozen tamod': 'Pornographic website', 'hindians': 'North Indian', 'hindian': 'North Indian',\n                  'celibatess': 'celibates', 'Trimp': 'Trump', 'wanket': 'wanker', 'wouldd': 'would',\n                  'arragent': 'arrogant', 'Ra - apist': 'rapist', 'idoot': 'idiot', 'gangstalkers': 'gangs talkers',\n                  'toastsexual': 'toast sexual', 'inapropriately': 'inappropriately', 'dumbassess': 'dumbass',\n                  'germanized': 'become german', 'helisexual': 'sexual', 'regilious': 'religious',\n                  'timetraveller': 'time traveller', 'darkwebcrawler': 'dark webcrawler', 'routez': 'route',\n                  'trumpians': 'Trump supporters', 'irreputable': 'reputation', 'serieusly': 'seriously',\n                  'anti cipation': 'anticipation', 'microaggression': 'micro aggression', 'Afircans': 'Africans',\n                  'microapologize': 'micro apologize', 'Vishnus': 'Vishnu', 'excritment': 'excitement',\n                  'disagreemen': 'disagreement', 'gujratis': 'gujarati', 'gujaratis': 'gujarati',\n                  'ugggggggllly': 'ugly',\n                  'Germanity': 'German', 'SoyBoys': 'cuck men lacking masculine characteristics',\n                  'н': 'h', 'м': 'm', 'ѕ': 's', 'т': 't', 'в': 'b', 'υ': 'u', 'ι': 'i',\n                  'genetilia': 'genitalia', 'r - apist': 'rapist', 'Borokabama': 'Barack Obama',\n                  'arectifier': 'rectifier', 'pettypotus': 'petty potus', 'magibabble': 'magi babble',\n                  'nothinking': 'thinking', 'centimiters': 'centimeters', 'saffronized': 'India, politics, derogatory',\n                  'saffronize': 'India, politics, derogatory', ' incect ': ' insect ', 'weenus': 'elbow skin',\n                  'Pakistainies': 'Pakistanis', 'goodspeaks': 'good speaks', 'inpregnated': 'in pregnant',\n                  'rapefilms': 'rape films', 'rapiest': 'rapist', 'hatrednesss': 'hatred',\n                  'heightism': 'height discrimination', 'getmy': 'get my', 'onsocial': 'on social',\n                  'worstplatform': 'worst platform', 'platfrom': 'platform', 'instagate': 'instigate',\n                  'Loy Machedeo': 'person', ' dsire ': ' desire ', 'iservant': 'servant', 'intelliegent': 'intelligent',\n                  'WW 1': ' WW1 ', 'WW 2': ' WW2 ', 'ww 1': ' WW1 ', 'ww 2': ' WW2 ',\n                  'keralapeoples': 'kerala peoples', 'trumpervotes': 'trumper votes', 'fucktrumpet': 'fuck trumpet',\n                  'likebJaish': 'like bJaish', 'likemy': 'like my', 'Howlikely': 'How likely',\n                  'disagreementts': 'disagreements', 'disagreementt': 'disagreement',\n                  'meninist': \"male chauvinism\", 'feminists': 'feminism supporters', 'Ghumendra': 'Bhupendra',\n                  'emellishments': 'embellishments',\n                  'settelemen': 'settlement',\n                  'Richmencupid': 'rich men dating website', 'richmencupid': 'rich men dating website',\n                  'Gaudry - Schost': '', 'ladymen': 'ladyboy', 'hasserment': 'Harassment',\n                  'instrumentalizing': 'instrument', 'darskin': 'dark skin', 'balckwemen': 'balck women',\n                  'recommendor': 'recommender', 'wowmen': 'women', 'expertthink': 'expert think',\n                  'whitesplaining': 'white splaining', 'Inquoraing': 'inquiring', 'whilemany': 'while many',\n                  'manyother': 'many other', 'involvedinthe': 'involved in the', 'slavetrade': 'slave trade',\n                  'aswell': 'as well', 'fewshowanyRemorse': 'few show any Remorse', 'trageting': 'targeting',\n                  'getile': 'gentile', 'Gujjus': 'derogatory Gujarati', 'judisciously': 'judiciously',\n                  'Hue Mungus': 'feminist bait', 'Hugh Mungus': 'feminist bait', 'Hindustanis': '',\n                  'Virushka': 'Great Relationships Couple', 'exclusinary': 'exclusionary', 'himdus': 'hindus',\n                  'Milo Yianopolous': 'a British polemicist', 'hidusim': 'hinduism',\n                  'holocaustable': 'holocaust', 'evangilitacal': 'evangelical', 'Busscas': 'Buscas',\n                  'holocaustal': 'holocaust', 'incestious': 'incestuous', 'Tennesseus': 'Tennessee',\n                  'GusDur': 'Gus Dur',\n                  'RPatah - Tan Eng Hwan': 'Silsilah', 'Reinfectus': 'reinfect', 'pharisaistic': 'pharisaism',\n                  'nuslims': 'Muslims', 'taskus': '', 'musims': 'Muslims',\n                  'Musevi': 'the independence of Mexico', ' racious ': 'discrimination expression of racism',\n                  'Muslimophobia': 'Muslim phobia', 'justyfied': 'justified', 'holocause': 'holocaust',\n                  'musilim': 'Muslim', 'misandrous': 'misandry', 'glrous': 'glorious', 'desemated': 'decimated',\n                  'votebanks': 'vote banks', 'Parkistan': 'Pakistan', 'Eurooe': 'Europe', 'animlaistic': 'animalistic',\n                  'Asiasoid': 'Asian', 'Congoid': 'Congolese', 'inheritantly': 'inherently',\n                  'Asianisation': 'Becoming Asia',\n                  'Russosphere': 'russia sphere of influence', 'exMuslims': 'Ex-Muslims',\n                  'discriminatein': 'discrimination', ' hinus ': ' hindus ', 'Nibirus': 'Nibiru',\n                  'habius - corpus': 'habeas corpus', 'prentious': 'pretentious', 'Sussia': 'ancient Jewish village',\n                  'moustachess': 'moustaches', 'Russions': 'Russians', 'Yuguslavia': 'Yugoslavia',\n                  'atrocitties': 'atrocities', 'Muslimophobe': 'Muslim phobic', 'fallicious': 'fallacious',\n                  'recussed': 'recursed', '@ usafmonitor': '', 'lustfly': 'lustful', 'canMuslims': 'can Muslims',\n                  'journalust': 'journalist', 'digustingly': 'disgustingly', 'harasing': 'harassing',\n                  'greatuncle': 'great uncle', 'Drumpf': 'Trump', 'rejectes': 'rejected', 'polyagamous': 'polygamous',\n                  'Mushlims': 'Muslims', 'accusition': 'accusation', 'geniusses': 'geniuses',\n                  'moustachesomething': 'moustache something', 'heineous': 'heinous',\n                  'Sapiosexuals': 'sapiosexual', 'sapiosexuals': 'sapiosexual', 'Sapiosexual': 'sapiosexual',\n                  'sapiosexual': 'Sexually attracted to intelligence', 'pansexuals': 'pansexual',\n                  'autosexual': 'auto sexual', 'sexualSlutty': 'sexual Slutty', 'hetorosexuality': 'hetoro sexuality',\n                  'chinesese': 'chinese', 'pizza gate': 'debunked conspiracy theory',\n                  'countryless': 'Having no country',\n                  'muslimare': 'Muslim are', 'iPhoneX': 'iPhone', 'lionese': 'lioness', 'marionettist': 'Marionettes',\n                  'demonetize': 'demonetized', 'eneyone': 'anyone', 'Karonese': 'Karo people Indonesia',\n                  'minderheid': 'minder worse', 'mainstreamly': 'mainstream', 'contraproductive': 'contra productive',\n                  'diffenky': 'differently', 'abandined': 'abandoned', 'p0 rnstars': 'pornstars',\n                  'overproud': 'over proud',\n                  'cheekboned': 'cheek boned', 'heriones': 'heroines', 'eventhogh': 'even though',\n                  'americanmedicalassoc': 'american medical assoc', 'feelwhen': 'feel when', 'Hhhow': 'how',\n                  'reallySemites': 'really Semites', 'gamergaye': 'gamersgate', 'manspreading': 'man spreading',\n                  'thammana': 'Tamannaah Bhatia', 'dogmans': 'dogmas', 'managementskills': 'management skills',\n                  'mangoliod': 'mongoloid', 'geerymandered': 'gerrymandered', 'mandateing': 'man dateing',\n                  'Romanium': 'Romanum',\n                  'mailwoman': 'mail woman', 'humancoalition': 'human coalition',\n                  'manipullate': 'manipulate', 'everyo0 ne': 'everyone', 'takeove': 'takeover',\n                  'Nonchristians': 'Non Christians', 'goverenments': 'governments', 'govrment': 'government',\n                  'polygomists': 'polygamists', 'Demogorgan': 'Demogorgon', 'maralago': 'Mar-a-Lago',\n                  'antibigots': 'anti bigots', 'gouing': 'going', 'muzaffarbad': 'muzaffarabad',\n                  'suchvstupid': 'such stupid', 'apartheidisrael': 'apartheid israel', \n                  'personaltiles': 'personal titles', 'lawyergirlfriend': 'lawyer girl friend',\n                  'northestern': 'northwestern', 'yeardold': 'years old', 'masskiller': 'mass killer',\n                  'southeners': 'southerners', 'Unitedstatesian': 'United states',\n\n                  'peoplekind': 'people kind', 'peoplelike': 'people like', 'countrypeople': 'country people',\n                  'shitpeople': 'shit people', 'trumpology': 'trump ology', 'trumpites': 'Trump supporters',\n                  'trumplies': 'trump lies', 'donaldtrumping': 'donald trumping', 'trumpdating': 'trump dating',\n                  'trumpsters': 'trumpeters', 'ciswomen': 'cis women', 'womenizer': 'womanizer',\n                  'pregnantwomen': 'pregnant women', 'autoliker': 'auto liker', 'smelllike': 'smell like',\n                  'autolikers': 'auto likers', 'religiouslike': 'religious like', 'likemail': 'like mail',\n                  'fislike': 'dislike', 'sneakerlike': 'sneaker like', 'like⬇': 'like',\n                  'likelovequotes': 'like lovequotes', 'likelogo': 'like logo', 'sexlike': 'sex like',\n                  'Whatwould': 'What would', 'Howwould': 'How would', 'manwould': 'man would',\n                  'exservicemen': 'ex servicemen', 'femenism': 'feminism', 'devopment': 'development',\n                  'doccuments': 'documents', 'supplementplatform': 'supplement platform', 'mendatory': 'mandatory',\n                  'moviments': 'movements', 'Kremenchuh': 'Kremenchug', 'docuements': 'documents',\n                  'determenism': 'determinism', 'envisionment': 'envision ment',\n                  'tricompartmental': 'tri compartmental', 'AddMovement': 'Add Movement',\n                  'mentionong': 'mentioning', 'Whichtreatment': 'Which treatment', 'repyament': 'repayment',\n                  'insemenated': 'inseminated', 'inverstment': 'investment',\n                  'managemental': 'manage mental', 'Inviromental': 'Environmental', 'menstrution': 'menstruation',\n                  'indtrument': 'instrument', 'mentenance': 'maintenance', 'fermentqtion': 'fermentation',\n                  'achivenment': 'achievement', 'mismanagements': 'mis managements', 'requriment': 'requirement',\n                  'denomenator': 'denominator', 'drparment': 'department', 'acumens': 'acumen s',\n                  'celemente': 'Clemente', 'manajement': 'management', 'govermenent': 'government',\n                  'accomplishmments': 'accomplishments', 'rendementry': 'rendement ry',\n                  'repariments': 'departments', 'menstrute': 'menstruate', 'determenistic': 'deterministic',\n                  'resigment': 'resignment', 'selfpayment': 'self payment', 'imrpovement': 'improvement',\n                  'enivironment': 'environment', 'compartmentley': 'compartment',\n                  'augumented': 'augmented', 'parmenent': 'permanent', 'dealignment': 'de alignment',\n                  'develepoments': 'developments', 'menstrated': 'menstruated', 'phnomenon': 'phenomenon',\n                  'Employmment': 'Employment', 'dimensionalise': 'dimensional ise', 'menigioma': 'meningioma',\n                  'recrument': 'recrement', 'Promenient': 'Provenient', 'gonverment': 'government',\n                  'statemment': 'statement', 'recuirement': 'requirement', 'invetsment': 'investment',\n                  'parilment': 'parchment', 'parmently': 'patiently', 'agreementindia': 'agreement india',\n                  'menifesto': 'manifesto', 'accomplsihments': 'accomplishments', 'disangagement': 'disengagement',\n                  'aevelopment': 'development', 'procument': 'procumbent', 'harashment': 'harassment',\n                  'Tiannanmen': 'Tiananmen', 'commensalisms': 'commensal isms', 'devlelpment': 'development',\n                  'dimensons': 'dimensions', 'recruitment2017': 'recruitment 2017', 'polishment': 'pol ishment',\n                  'CommentSafe': 'Comment Safe', 'meausrements': 'measurements', 'geomentrical': 'geometrical',\n                  'undervelopment': 'undevelopment', 'mensurational': 'mensuration al', 'fanmenow': 'fan menow',\n                  'permenganate': 'permanganate', 'bussinessmen': 'businessmen',\n                  'supertournaments': 'super tournaments', 'permanmently': 'permanently',\n                  'lamenectomy': 'lamnectomy', 'assignmentcanyon': 'assignment canyon', 'adgestment': 'adjustment',\n                  'mentalized': 'metalized', 'docyments': 'documents', 'requairment': 'requirement',\n                  'batsmencould': 'batsmen could', 'argumentetc': 'argument etc', 'enjoiment': 'enjoyment',\n                  'invement': 'movement', 'accompliushments': 'accomplishments', 'regements': 'regiments',\n                  'departmentHow': 'department How', 'Aremenian': 'Armenian', 'amenclinics': 'amen clinics',\n                  'nonfermented': 'non fermented', 'Instumentation': 'Instrumentation', 'mentalitiy': 'mentality',\n                  ' govermen ': 'goverment', 'underdevelopement': 'under developement', 'parlimentry': 'parliamentary',\n                  'indemenity': 'indemnity', 'Inatrumentation': 'Instrumentation', 'menedatory': 'mandatory',\n                  'mentiri': 'entire', 'accomploshments': 'accomplishments', 'instrumention': 'instrument ion',\n                  'afvertisements': 'advertisements', 'parlementarian': 'parlement arian',\n                  'entitlments': 'entitlements', 'endrosment': 'endorsement', 'improment': 'impriment',\n                  'archaemenid': 'Achaemenid', 'replecement': 'replacement', 'placdment': 'placement',\n                  'femenise': 'feminise', 'envinment': 'environment', 'AmenityCompany': 'Amenity Company',\n                  'increaments': 'increments', 'accomplihsments': 'accomplishments',\n                  'manygovernment': 'many government', 'panishments': 'punishments', 'elinment': 'eloinment',\n                  'mendalin': 'mend alin', 'farmention': 'farm ention', 'preincrement': 'pre increment',\n                  'postincrement': 'post increment', 'achviements': 'achievements', 'menditory': 'mandatory',\n                  'Emouluments': 'Emoluments', 'Stonemen': 'Stone men', 'menmium': 'medium',\n                  'entaglement': 'entanglement', 'integumen': 'integument', 'harassument': 'harassment',\n                  'retairment': 'retainment', 'enviorement': 'environment', 'tormentous': 'torment ous',\n                  'confiment': 'confident', 'Enchroachment': 'Encroachment', 'prelimenary': 'preliminary',\n                  'fudamental': 'fundamental', 'instrumenot': 'instrument', 'icrement': 'increment',\n                  'prodimently': 'prominently', 'meniss': 'menise', 'Whoimplemented': 'Who implemented',\n                  'Representment': 'Rep resentment', 'StartFragment': 'Start Fragment',\n                  'EndFragment': 'End Fragment', ' documentarie ': ' documentaries ', 'requriments': 'requirements',\n                  'constitutionaldevelopment': 'constitutional development', 'parlamentarians': 'parliamentarians',\n                  'Rumenova': 'Rumen ova', 'argruments': 'arguments', 'findamental': 'fundamental',\n                  'totalinvestment': 'total investment', 'gevernment': 'government', 'recmommend': 'recommend',\n                  'appsmoment': 'apps moment', 'menstruual': 'menstrual', 'immplemented': 'implemented',\n                  'engangement': 'engagement', 'invovement': 'involvement', 'returement': 'retirement',\n                  'simentaneously': 'simultaneously', 'accompishments': 'accomplishments',\n                  'menstraution': 'menstruation', 'experimently': 'experiment', 'abdimen': 'abdomen',\n                  'cemenet': 'cement', 'propelment': 'propel ment', 'unamendable': 'un amendable',\n                  'employmentnews': 'employment news', 'lawforcement': 'law forcement',\n                  'menstuating': 'menstruating', 'fevelopment': 'development', 'reglamented': 'reg lamented',\n                  'imrovment': 'improvement', 'recommening': 'recommending', 'sppliment': 'supplement',\n                  'measument': 'measurement', 'reimbrusement': 'reimbursement', 'Nutrament': 'Nutriment',\n                  'puniahment': 'punishment', 'subligamentous': 'sub ligamentous', 'comlementry': 'complementary',\n                  'reteirement': 'retirement', 'envioronments': 'environments', 'haraasment': 'harassment',\n                  'USAgovernment': 'USA government', 'Apartmentfinder': 'Apartment finder',\n                  'encironment': 'environment', 'metacompartment': 'meta compartment',\n                  'augumentation': 'argumentation', 'dsymenorrhoea': 'dysmenorrhoea',\n                  'nonabandonment': 'non abandonment', 'annoincement': 'announcement',\n                  'menberships': 'memberships', 'Gamenights': 'Game nights', 'enliightenment': 'enlightenment',\n                  'supplymentry': 'supplementary', 'parlamentary': 'parliamentary', 'duramen': 'dura men',\n                  'hotelmanagement': 'hotel management', 'deartment': 'department',\n                  'treatmentshelp': 'treatments help', 'attirements': 'attire ments',\n                  'amendmending': 'amend mending', 'pseudomeningocele': 'pseudo meningocele',\n                  'intrasegmental': 'intra segmental', 'treatmenent': 'treatment', 'infridgement': 'infringement',\n                  'infringiment': 'infringement', 'recrecommend': 'rec recommend', 'entartaiment': 'entertainment',\n                  'inplementing': 'implementing', 'indemendent': 'independent', 'tremendeous': 'tremendous',\n                  'commencial': 'commercial', 'scomplishments': 'accomplishments', 'Emplement': 'Implement',\n                  'dimensiondimensions': 'dimension dimensions', 'depolyment': 'deployment',\n                  'conpartment': 'compartment', 'govnments': 'movements', 'menstrat': 'menstruate',\n                  'accompplishments': 'accomplishments', 'Enchacement': 'Enchancement',\n                  'developmenent': 'development', 'emmenagogues': 'emmenagogue', 'aggeement': 'agreement',\n                  'elementsbond': 'elements bond', 'remenant': 'remnant', 'Manamement': 'Management',\n                  'Augumented': 'Augmented', 'dimensonless': 'dimensionless',\n                  'ointmentsointments': 'ointments ointments', 'achiements': 'achievements',\n                  'recurtment': 'recurrent', 'gouverments': 'governments', 'docoment': 'document',\n                  'programmingassignments': 'programming assignments', 'menifest': 'manifest',\n                  'investmentguru': 'investment guru', 'deployements': 'deployments', 'Invetsment': 'Investment',\n                  'plaement': 'placement', 'Perliament': 'Parliament', 'femenists': 'feminists',\n                  'ecumencial': 'ecumenical', 'advamcements': 'advancements', 'refundment': 'refund ment',\n                  'settlementtake': 'settlement take', 'mensrooms': 'mens rooms',\n                  'productManagement': 'product Management', 'armenains': 'armenians',\n                  'betweenmanagement': 'between management', 'difigurement': 'disfigurement',\n                  'Armenized': 'Armenize', 'hurrasement': 'hurra sement', 'mamgement': 'management',\n                  'momuments': 'monuments', 'eauipments': 'equipments', 'managemenet': 'management',\n                  'treetment': 'treatment', 'webdevelopement': 'web developement', 'supplemenary': 'supplementary',\n                  'Encironmental': 'Environmental', 'Understandment': 'Understand ment',\n                  'enrollnment': 'enrollment', 'thinkstrategic': 'think strategic', 'thinkinh': 'thinking',\n                  'Softthinks': 'Soft thinks', 'underthinking': 'under thinking', 'thinksurvey': 'think survey',\n                  'whitelash': 'white lash', 'whiteheds': 'whiteheads', 'whitetning': 'whitening',\n                  'whitegirls': 'white girls', 'whitewalkers': 'white walkers', 'manycountries': 'many countries',\n                  'accomany': 'accompany', 'fromGermany': 'from Germany', 'manychat': 'many chat',\n                  'Germanyl': 'Germany l', 'manyness': 'many ness', 'many4': 'many', 'exmuslims': 'ex muslims',\n                  'digitizeindia': 'digitize india', 'indiarush': 'india rush', 'indiareads': 'india reads',\n                  'telegraphindia': 'telegraph india', 'Southindia': 'South india', 'Airindia': 'Air india',\n                  'siliconindia': 'silicon india', 'airindia': 'air india', 'indianleaders': 'indian leaders',\n                  'fundsindia': 'funds india', 'indianarmy': 'indian army', 'Technoindia': 'Techno india',\n                  'Betterindia': 'Better india', 'capesindia': 'capes india', 'Rigetti': 'Ligetti',\n                  'vegetablr': 'vegetable', 'get90': 'get', 'Magetta': 'Maretta', 'nagetive': 'native',\n                  'isUnforgettable': 'is Unforgettable', 'get630': 'get 630', 'GadgetPack': 'Gadget Pack',\n                  'Languagetool': 'Language tool', 'bugdget': 'budget', 'africaget': 'africa get',\n                  'ABnegetive': 'Abnegative', 'orangetheory': 'orange theory', 'getsmuggled': 'get smuggled',\n                  'avegeta': 'ave geta', 'gettubg': 'getting', 'gadgetsnow': 'gadgets now',\n                  'surgetank': 'surge tank', 'gadagets': 'gadgets', 'getallparts': 'get allparts',\n                  'messenget': 'messenger', 'vegetarean': 'vegetarian', 'get1000': 'get 1000',\n                  'getfinancing': 'get financing', 'getdrip': 'get drip', 'AdsTargets': 'Ads Targets',\n                  'tgethr': 'together', 'vegetaries': 'vegetables', 'forgetfulnes': 'forgetfulness',\n                  'fisgeting': 'fidgeting', 'BudgetAir': 'Budget Air',\n                  'getDepersonalization': 'get Depersonalization', 'negetively': 'negatively',\n                  'gettibg': 'getting', 'nauget': 'naught', 'Bugetti': 'Bugatti', 'plagetum': 'plage tum',\n                  'vegetabale': 'vegetable', 'changetip': 'change tip', 'blackwashing': 'black washing',\n                  'blackpink': 'black pink', 'blackmoney': 'black money',\n                  'blackmarks': 'black marks', 'blackbeauty': 'black beauty', 'unblacklisted': 'un blacklisted',\n                  'blackdotes': 'black dotes', 'blackboxing': 'black boxing', 'blackpaper': 'black paper',\n                  'blackpower': 'black power', 'Latinamericans': 'Latin americans', 'musigma': 'mu sigma',\n                  'Indominus': 'In dominus', 'usict': 'USSCt', 'indominus': 'in dominus', 'Musigma': 'Mu sigma',\n                  'plus5': 'plus', 'Russiagate': 'Russia gate', 'russophobic': 'Russophobiac',\n                  'Marcusean': 'Marcuse an', 'Radijus': 'Radius', 'cobustion': 'combustion',\n                  'Austrialians': 'Australians', 'mylogenous': 'myogenous', 'Raddus': 'Radius',\n                  'hetrogenous': 'heterogenous', 'greenhouseeffect': 'greenhouse effect', 'aquous': 'aqueous',\n                  'Taharrush': 'Tahar rush', 'Senousa': 'Venous', 'diplococcus': 'diplo coccus',\n                  'CityAirbus': 'City Airbus', 'sponteneously': 'spontaneously', 'trustless': 't rustless',\n                  'Pushkaram': 'Pushkara m', 'Fusanosuke': 'Fu sanosuke', 'isthmuses': 'isthmus es',\n                  'lucideus': 'lucidum', 'overjustification': 'over justification', 'Bindusar': 'Bind usar',\n                  'cousera': 'couler', 'musturbation': 'masturbation', 'infustry': 'industry',\n                  'Huswifery': 'Huswife ry', 'rombous': 'bombous', 'disengenuously': 'disingenuously',\n                  'sllybus': 'syllabus', 'celcious': 'delicious', 'cellsius': 'celsius',\n                  'lethocerus': 'Lethocerus', 'monogmous': 'monogamous', 'Ballyrumpus': 'Bally rumpus',\n                  'Koushika': 'Koushik a', 'vivipoarous': 'viviparous', 'ludiculous': 'ridiculous',\n                  'sychronous': 'synchronous', 'industiry': 'industry', 'scuduse': 'scud use',\n                  'babymust': 'baby must', 'simultqneously': 'simultaneously', 'exust': 'ex ust',\n                  'notmusing': 'not musing', 'Zamusu': 'Amuse', 'tusaki': 'tu saki', 'Marrakush': 'Marrakesh',\n                  'justcheaptickets': 'just cheaptickets', 'Ayahusca': 'Ayahausca', 'samousa': 'samosa',\n                  'Gusenberg': 'Gutenberg', 'illustratuons': 'illustrations', 'extemporeneous': 'extemporaneous',\n                  'Mathusla': 'Mathusala', 'Confundus': 'Con fundus', 'tusts': 'trusts', 'poisenious': 'poisonous',\n                  'Mevius': 'Medius', 'inuslating': 'insulating', 'aroused21000': 'aroused 21000',\n                  'Wenzeslaus': 'Wenceslaus', 'JustinKase': 'Justin Kase', 'purushottampur': 'purushottam pur',\n                  'citruspay': 'citrus pay', 'secutus': 'sects', 'austentic': 'austenitic',\n                  'FacePlusPlus': 'Face PlusPlus', 'aysnchronous': 'asynchronous',\n                  'teamtreehouse': 'team treehouse', 'uncouncious': 'unconscious', 'Priebuss': 'Prie buss',\n                  'consciousuness': 'consciousness', 'susubsoil': 'su subsoil', 'trimegistus': 'Trismegistus',\n                  'protopeterous': 'protopterous', 'trustworhty': 'trustworthy', 'ushually': 'usually',\n                  'industris': 'industries', 'instantneous': 'instantaneous', 'superplus': 'super plus',\n                  'shrusti': 'shruti', 'hindhus': 'hindus', 'outonomous': 'autonomous', 'reliegious': 'religious',\n                  'Kousakis': 'Kou sakis', 'reusult': 'result', 'JanusGraph': 'Janus Graph',\n                  'palusami': 'palus ami', 'mussraff': 'muss raff', 'hukous': 'humous',\n                  'photoacoustics': 'photo acoustics', 'kushanas': 'kusha nas', 'justdile': 'justice',\n                  'Massahusetts': 'Massachusetts', 'uspset': 'upset', 'sustinet': 'sustinent',\n                  'consicious': 'conscious', 'Sadhgurus': 'Sadh gurus', 'hystericus': 'hysteric us',\n                  'visahouse': 'visa house', 'supersynchronous': 'super synchronous', 'posinous': 'rosinous',\n                  'Fernbus': 'Fern bus', 'Tiltbrush': 'Tilt brush', 'glueteus': 'gluteus', 'posionus': 'poisons',\n                  'Freus': 'Frees', 'Zhuchengtyrannus': 'Zhucheng tyrannus', 'savonious': 'sanious',\n                  'CusJo': 'Cusco', 'congusion': 'confusion', 'dejavus': 'dejavu s', 'uncosious': 'uncopious',\n                  'previius': 'previous', 'counciousness': 'conciousness', 'lustorus': 'lustrous',\n                  'sllyabus': 'syllabus', 'mousquitoes': 'mosquitoes', 'Savvius': 'Savvies', 'arceius': 'Arcesius',\n                  'prejusticed': 'prejudiced', 'requsitioned': 'requisitioned',\n                  'deindustralization': 'deindustrialization', 'muscleblaze': 'muscle blaze',\n                  'ConsciousX5': 'conscious', 'nitrogenious': 'nitrogenous', 'mauritious': 'mauritius',\n                  'rigrously': 'rigorously', 'Yutyrannus': 'Yu tyrannus', 'muscualr': 'muscular',\n                  'conscoiusness': 'consciousness', 'Causians': 'Crusians', 'WorkFusion': 'Work Fusion',\n                  'puspak': 'pu spak', 'Inspirus': 'Inspires', 'illiustrations': 'illustrations',\n                  'Nobushi': 'No bushi', 'theuseof': 'thereof', 'suspicius': 'suspicious', 'Intuous': 'Virtuous',\n                  'gaushalas': 'gaus halas', 'campusthrough': 'campus through', 'seriousity': 'seriosity',\n                  'resustence': 'resistence', 'geminatus': 'geminates', 'disquss': 'discuss',\n                  'nicholus': 'nicholas', 'Husnai': 'Hussar', 'diiscuss': 'discuss', 'diffussion': 'diffusion',\n                  'phusicist': 'physicist', 'ernomous': 'enormous', 'Khushali': 'Khushal i', 'heitus': 'Leitus',\n                  'cracksbecause': 'cracks because', 'Nautlius': 'Nautilus', 'trausted': 'trusted',\n                  'Dardandus': 'Dardanus', 'Megatapirus': 'Mega tapirus', 'clusture': 'culture',\n                  'vairamuthus': 'vairamuthu s', 'disclousre': 'disclosure',\n                  'industrilaization': 'industrialization', 'musilms': 'muslims', 'Australia9': 'Australian',\n                  'causinng': 'causing', 'ibdustries': 'industries', 'searious': 'serious',\n                  'Coolmuster': 'Cool muster', 'sissyphus': 'sisyphus', ' justificatio ': 'justification',\n                  'antihindus': 'anti hindus', 'Moduslink': 'Modus link', 'zymogenous': 'zymogen ous',\n                  'prospeorus': 'prosperous', 'Retrocausality': 'Retro causality', 'FusionGPS': 'Fusion GPS',\n                  'Mouseflow': 'Mouse flow', 'bootyplus': 'booty plus', 'Itylus': 'I tylus',\n                  'Olnhausen': 'Olshausen', 'suspeect': 'suspect', 'entusiasta': 'enthusiast',\n                  'fecetious': 'facetious', 'bussiest': 'fussiest', 'Draconius': 'Draconis',\n                  'requsite': 'requisite', 'nauseatic': 'nausea tic', 'Brusssels': 'Brussels',\n                  'repurcussion': 'repercussion', 'Jeisus': 'Jesus', 'philanderous': 'philander ous',\n                  'muslisms': 'muslims', 'august2017': 'august 2017', 'calccalculus': 'calc calculus',\n                  'unanonymously': 'un anonymously', 'Imaprtus': 'Impetus', 'carnivorus': 'carnivorous',\n                  'Corypheus': 'Coryphees', 'austronauts': 'astronauts', 'neucleus': 'nucleus',\n                  'housepoor': 'house poor', 'rescouses': 'responses', 'Tagushi': 'Tagus hi',\n                  'hyperfocusing': 'hyper focusing', 'nutriteous': 'nutritious', 'chylus': 'chylous',\n                  'preussure': 'pressure', 'outfocus': 'out focus', 'Hanfus': 'Hannus', 'Rustyrose': 'Rusty rose',\n                  'vibhushant': 'vibhushan t', 'conciousnes': 'conciousness', 'Venus25': 'Venus',\n                  'Sedataious': 'Seditious', 'promuslim': 'pro muslim', 'statusGuru': 'status Guru',\n                  'yousician': 'musician', 'transgenus': 'trans genus', 'Pushbullet': 'Push bullet',\n                  'jeesyllabus': 'jee syllabus', 'complusary': 'compulsory', 'Holocoust': 'Holocaust',\n                  'careerplus': 'career plus', 'Lllustrate': 'Illustrate', 'Musino': 'Musion',\n                  'Phinneus': 'Phineus', 'usedtoo': 'used too', 'JustBasic': 'Just Basic', 'webmusic': 'web music',\n                  'TrustKit': 'Trust Kit', 'industrZgies': 'industries', 'rubustness': 'robustness',\n                  'Missuses': 'Miss uses', 'Musturbation': 'Masturbation', 'bustees': 'bus tees',\n                  'justyfy': 'justify', 'pegusus': 'pegasus', 'industrybuying': 'industry buying',\n                  'advantegeous': 'advantageous', 'kotatsus': 'kotatsu s', 'justcreated': 'just created',\n                  'simultameously': 'simultaneously', 'husoone': 'huso one', 'twiceusing': 'twice using',\n                  'cetusplay': 'cetus play', 'sqamous': 'squamous', 'claustophobic': 'claustrophobic',\n                  'Kaushika': 'Kaushik a', 'dioestrus': 'di oestrus', 'Degenerous': 'De generous',\n                  'neculeus': 'nucleus', 'cutaneously': 'cu taneously', 'Alamotyrannus': 'Alamo tyrannus',\n                  'Ivanious': 'Avanious', 'arceous': 'araceous', 'Flixbus': 'Flix bus', 'caausing': 'causing',\n                  'publious': 'Publius', 'Juilus': 'Julius', 'Australianism': 'Australian ism',\n                  'vetronus': 'verrons', 'nonspontaneous': 'non spontaneous', 'calcalus': 'calculus',\n                  'commudus': 'Commodus', 'Rheusus': 'Rhesus', 'syallubus': 'syllabus', 'Yousician': 'Musician',\n                  'qurush': 'qu rush', 'athiust': 'athirst', 'conclusionless': 'conclusion less',\n                  'usertesting': 'user testing', 'redius': 'radius', 'Austrolia': 'Australia',\n                  'sllaybus': 'syllabus', 'toponymous': 'top onymous', 'businiss': 'business',\n                  'hyperthalamus': 'hyper thalamus', 'clause55': 'clause', 'cosicous': 'conscious',\n                  'Sushena': 'Saphena', 'Luscinus': 'Luscious', 'Prussophile': 'Russophile', 'jeaslous': 'jealous',\n                  'Austrelia': 'Australia', 'contiguious': 'contiguous',\n                  'subconsciousnesses': 'sub consciousnesses', ' jusification ': 'justification',\n                  'dilusion': 'delusion', 'anticoncussive': 'anti concussive', 'disngush': 'disgust',\n                  'constiously': 'consciously', 'filabustering': 'filibustering', 'GAPbuster': 'GAP buster',\n                  'insectivourous': 'insectivorous', 'glocuse': 'louse', 'Antritrust': 'Antitrust',\n                  'thisAustralian': 'this Australian', 'FusionDrive': 'Fusion Drive', 'nuclus': 'nucleus',\n                  'abussive': 'abusive', 'mustang1': 'mustangs', 'inradius': 'in radius', 'polonious': 'polonius',\n                  'ofKulbhushan': 'of Kulbhushan', 'homosporous': 'homos porous', 'circumradius': 'circum radius',\n                  'atlous': 'atrous', 'insustry': 'industry', 'campuswith': 'campus with', 'beacsuse': 'because',\n                  'concuous': 'conscious', 'nonHindus': 'non Hindus', 'carnivourous': 'carnivorous',\n                  'tradeplus': 'trade plus', 'Jeruselam': 'Jerusalem',\n                  'musuclar': 'muscular', 'deangerous': 'dangerous', 'disscused': 'discussed',\n                  'industdial': 'industrial', 'sallatious': 'fallacious', 'rohmbus': 'rhombus',\n                  'golusu': 'gol usu', 'Minangkabaus': 'Minangkabau s', 'Mustansiriyah': 'Mustansiriya h',\n                  'anomymously': 'anonymously', 'abonymously': 'anonymously', 'indrustry': 'industry',\n                  'Musharrf': 'Musharraf', 'workouses': 'workhouses', 'sponataneously': 'spontaneously',\n                  'anmuslim': 'an muslim', 'syallbus': 'syllabus', 'presumptuousnes': 'presumptuousness',\n                  'Thaedus': 'Thaddus', 'industey': 'industry', 'hkust': 'hust', 'Kousseri': 'Kousser i',\n                  'mousestats': 'mouses tats', 'russiagate': 'russia gate', 'simantaneously': 'simultaneously',\n                  'Austertana': 'Auster tana', 'infussions': 'infusions', 'coclusion': 'conclusion',\n                  'sustainabke': 'sustainable', 'tusami': 'tu sami', 'anonimously': 'anonymously',\n                  'usebase': 'use base', 'balanoglossus': 'Balanoglossus', 'Unglaus': 'Ung laus',\n                  'ignoramouses': 'ignoramuses', 'snuus': 'snugs', 'reusibility': 'reusability',\n                  'Straussianism': 'Straussian ism', 'simoultaneously': 'simultaneously',\n                  'realbonus': 'real bonus', 'nuchakus': 'nunchakus', 'annonimous': 'anonymous',\n                  'Incestious': 'Incestuous', 'Manuscriptology': 'Manuscript ology', 'difusse': 'diffuse',\n                  'Pliosaurus': 'Pliosaur us', 'cushelle': 'cush elle', 'Catallus': 'Catullus',\n                  'MuscleBlaze': 'Muscle Blaze', 'confousing': 'confusing', 'enthusiasmless': 'enthusiasm less',\n                  'Tetherusd': 'Tethered', 'Josephius': 'Josephus', 'jusrlt': 'just',\n                  'simutaneusly': 'simultaneously', 'mountaneous': 'mountainous', 'Badonicus': 'Sardonicus',\n                  'muccus': 'mucous', 'nicus': 'nidus', 'austinlizards': 'austin lizards',\n                  'errounously': 'erroneously', 'Australua': 'Australia', 'sylaabus': 'syllabus',\n                  'dusyant': 'distant', 'javadiscussion': 'java discussion', 'megabuses': 'mega buses',\n                  'danergous': 'dangerous', 'contestious': 'contentious', 'exause': 'excuse',\n                  'muscluar': 'muscular', 'avacous': 'vacuous', 'Ingenhousz': 'Ingenious',\n                  'holocausting': 'holocaust ing', 'Pakustan': 'Pakistan', 'purusharthas': 'purushartha',\n                  'bapus': 'bapu s', 'useul': 'useful', 'pretenious': 'pretentious', 'homogeneus': 'homogeneous',\n                  'bhlushes': 'blushes', 'Saggittarius': 'Sagittarius', 'sportsusa': 'sports usa',\n                  'kerataconus': 'keratoconus', 'infrctuous': 'infectuous', 'Anonoymous': 'Anonymous',\n                  'triphosphorus': 'tri phosphorus', 'ridicjlously': 'ridiculously',\n                  'worldbusiness': 'world business', 'hollcaust': 'holocaust', 'Dusra': 'Dura',\n                  'meritious': 'meritorious', 'Sauskes': 'Causes', 'inudustry': 'industry',\n                  'frustratd': 'frustrate', 'hypotenous': 'hypogenous', 'Dushasana': 'Dush asana',\n                  'saadus': 'status', 'keratokonus': 'keratoconus', 'Jarrus': 'Harrus', 'neuseous': 'nauseous',\n                  'simutanously': 'simultaneously', 'diphosphorus': 'di phosphorus', 'sulprus': 'surplus',\n                  'Hasidus': 'Hasid us', 'suspenive': 'suspensive', 'illlustrator': 'illustrator',\n                  'userflows': 'user flows', 'intrusivethoughts': 'intrusive thoughts', 'countinous': 'continuous',\n                  'gpusli': 'gusli', 'Calculus1': 'Calculus', 'bushiri': 'Bushire',\n                  'torvosaurus': 'Torosaurus', 'chestbusters': 'chest busters', 'Satannus': 'Sat annus',\n                  'falaxious': 'fallacious', 'obnxious': 'obnoxious', 'tranfusions': 'transfusions',\n                  'PlayMagnus': 'Play Magnus', 'Epicodus': 'Episodes', 'Hypercubus': 'Hypercubes',\n                  'Musickers': 'Musick ers', 'programmebecause': 'programme because', 'indiginious': 'indigenous',\n                  'housban': 'Housman', 'iusso': 'kusso', 'annilingus': 'anilingus', 'Nennus': 'Genius',\n                  'pussboy': 'puss boy', 'Photoacoustics': 'Photo acoustics', 'Hindusthanis': 'Hindustanis',\n                  'lndustrial': 'industrial', 'tyrannously': 'tyrannous', 'Susanoomon': 'Susanoo mon',\n                  'colmbus': 'columbus', 'sussessful': 'successful', 'ousmania': 'ous mania',\n                  'ilustrating': 'illustrating', 'famousbirthdays': 'famous birthdays',\n                  'suspectance': 'suspect ance', 'extroneous': 'extraneous', 'teethbrush': 'teeth brush',\n                  'abcmouse': 'abc mouse', 'degenerous': 'de generous', 'doesGauss': 'does Gauss',\n                  'insipudus': 'insipidus', 'movielush': 'movie lush', 'Rustichello': 'Rustic hello',\n                  'Firdausiya': 'Firdausi ya', 'checkusers': 'check users', 'householdware': 'household ware',\n                  'prosporously': 'prosperously', 'SteLouse': 'Ste Louse', 'obfuscaton': 'obfuscation',\n                  'amorphus': 'amorph us', 'trustworhy': 'trustworthy', 'celsious': 'cesious',\n                  'dangorous': 'dangerous', 'anticancerous': 'anti cancerous', 'cousi ': 'cousin ',\n                  'austroloid': 'australoid', 'fergussion': 'percussion', 'andKyokushin': 'and Kyokushin',\n                  'cousan': 'cousin', 'Huskystar': 'Hu skystar', 'retrovisus': 'retrovirus', 'becausr': 'because',\n                  'Jerusalsem': 'Jerusalem', 'motorious': 'notorious', 'industrilised': 'industrialised',\n                  'powerballsusa': 'powerballs usa', 'monoceious': 'monoecious', 'batteriesplus': 'batteries plus',\n                  'nonviscuous': 'nonviscous', 'industion': 'induction', 'bussinss': 'bussings',\n                  'userbags': 'user bags', 'Jlius': 'Julius', 'thausand': 'thousand', 'plustwo': 'plus two',\n                  'defpush': 'def push', 'subconcussive': 'sub concussive', 'muslium': 'muslim',\n                  'industrilization': 'industrialization', 'Maurititus': 'Mauritius', 'uslme': 'some',\n                  'Susgaon': 'Surgeon', 'Pantherous': 'Panther ous', 'antivirius': 'antivirus',\n                  'Trustclix': 'Trust clix', 'silumtaneously': 'simultaneously', 'Icompus': 'Corpus',\n                  'atonomous': 'autonomous', 'Reveuse': 'Reve use', 'legumnous': 'leguminous',\n                  'syllaybus': 'syllabus', 'louspeaker': 'loudspeaker', 'susbtraction': 'substraction',\n                  'virituous': 'virtuous', 'disastrius': 'disastrous', 'jerussalem': 'jerusalem',\n                  'Industrailzed': 'Industrialized', 'recusion': 'recushion',\n                  'simultenously': 'simultaneously',\n                  'Pulphus': 'Pulpous', 'harbaceous': 'herbaceous', 'phlegmonous': 'phlegmon ous', 'use38': 'use',\n                  'jusify': 'justify', 'instatanously': 'instantaneously', 'tetramerous': 'tetramer ous',\n                  'usedvin': 'used vin', 'sagittarious': 'sagittarius', 'mausturbate': 'masturbate',\n                  'subcautaneous': 'subcutaneous', 'dangergrous': 'dangerous', 'sylabbus': 'syllabus',\n                  'hetorozygous': 'heterozygous', 'Ignasius': 'Ignacius', 'businessbor': 'business bor',\n                  'Bhushi': 'Thushi', 'Moussolini': 'Mussolini', 'usucaption': 'usu caption',\n                  'Customzation': 'Customization', 'cretinously': 'cretinous', 'genuiuses': 'geniuses',\n                  'Moushmee': 'Mousmee', 'neigous': 'nervous',\n                  'infrustructre': 'infrastructure', 'Ilusha': 'Ilesha', 'suconciously': 'unconciously',\n                  'stusy': 'study', 'mustectomy': 'mastectomy', 'Farmhousebistro': 'Farmhouse bistro',\n                  'instantanous': 'instantaneous', 'JustForex': 'Just Forex', 'Indusyry': 'Industry',\n                  'mustabating': 'must abating', 'uninstrusive': 'unintrusive', 'customshoes': 'customs hoes',\n                  'homageneous': 'homogeneous', 'Empericus': 'Imperious', 'demisexuality': 'demi sexuality',\n                  'transexualism': 'transsexualism', 'sexualises': 'sexualise', 'demisexuals': 'demisexual',\n                  'sexuly': 'sexily', 'Pornosexuality': 'Porno sexuality', 'sexond': 'second', 'sexxual': 'sexual',\n                  'asexaul': 'asexual', 'sextactic': 'sex tactic', 'sexualityism': 'sexuality ism',\n                  'monosexuality': 'mono sexuality', 'intwrsex': 'intersex', 'hypersexualize': 'hyper sexualize',\n                  'homosexualtiy': 'homosexuality', 'examsexams': 'exams exams', 'sexmates': 'sex mates',\n                  'sexyjobs': 'sexy jobs', 'sexitest': 'sexiest', 'fraysexual': 'fray sexual',\n                  'sexsurrogates': 'sex surrogates', 'sexuallly': 'sexually', 'gamersexual': 'gamer sexual',\n                  'greysexual': 'grey sexual', 'omnisexuality': 'omni sexuality', 'hetereosexual': 'heterosexual',\n                  'productsexamples': 'products examples', 'sexgods': 'sex gods', 'semisexual': 'semi sexual',\n                  'homosexulity': 'homosexuality', 'sexeverytime': 'sex everytime', 'neurosexist': 'neuro sexist',\n                  'worldquant': 'world quant', 'Freshersworld': 'Freshers world', 'smartworld': 'sm artworld',\n                  'Mistworlds': 'Mist worlds', 'boothworld': 'booth world', 'ecoworld': 'eco world',\n                  'Ecoworld': 'Eco world', 'underworldly': 'under worldly', 'worldrank': 'world rank',\n                  'Clearworld': 'Clear world', 'Boothworld': 'Booth world', 'Rimworld': 'Rim world',\n                  'cryptoworld': 'crypto world', 'machineworld': 'machine world', 'worldwideley': 'worldwide ley',\n                  'capuletwant': 'capulet want', 'Bhagwanti': 'Bhagwant i', 'Unwanted72': 'Unwanted 72',\n                  'wantrank': 'want rank',\n                  'willhappen': 'will happen', 'thateasily': 'that easily',\n                  'Whatevidence': 'What evidence', 'metaphosphates': 'meta phosphates',\n                  'exilarchate': 'exilarch ate', 'aulphate': 'sulphate', 'Whateducation': 'What education',\n                  'persulphates': 'per sulphates', 'disulphate': 'di sulphate', 'picosulphate': 'pico sulphate',\n                  'tetraosulphate': 'tetrao sulphate', 'prechinese': 'pre chinese',\n                  'Hellochinese': 'Hello chinese', 'muchdeveloped': 'much developed', 'stomuch': 'stomach',\n                  'Whatmakes': 'What makes', 'Lensmaker': 'Lens maker', 'eyemake': 'eye make',\n                  'Techmakers': 'Tech makers', 'cakemaker': 'cake maker', 'makeup411': 'makeup 411',\n                  'objectmake': 'object make', 'crazymaker': 'crazy maker', 'techmakers': 'tech makers',\n                  'makedonian': 'macedonian', 'makeschool': 'make school', 'anxietymake': 'anxiety make',\n                  'makeshifter': 'make shifter', 'countryball': 'country ball', 'Whichcountry': 'Which country',\n                  'countryHow': 'country How', 'Zenfone': 'Zen fone', 'Electroneum': 'Electro neum',\n                  'electroneum': 'electro neum', 'Demonetisation': 'demonetization', 'zenfone': 'zen fone',\n                  'ZenFone': 'Zen Fone', 'onecoin': 'one coin', 'demonetizing': 'demonetized',\n                  'iphone7': 'iPhone', 'iPhone6': 'iPhone', 'microneedling': 'micro needling', 'iphone6': 'iPhone',\n                  'Monegasques': 'Monegasque s', 'demonetised': 'demonetized',\n                  'EveryoneDiesTM': 'EveryoneDies TM', 'teststerone': 'testosterone', 'DoneDone': 'Done Done',\n                  'papermoney': 'paper money', 'Sasabone': 'Sasa bone', 'Blackphone': 'Black phone',\n                  'Bonechiller': 'Bone chiller', 'Moneyfront': 'Money front', 'workdone': 'work done',\n                  'iphoneX': 'iPhone', 'roxycodone': 'r oxycodone',\n                  'moneycard': 'money card', 'Fantocone': 'Fantocine', 'eletronegativity': 'electronegativity',\n                  'mellophones': 'mellophone s', 'isotones': 'iso tones', 'donesnt': 'doesnt',\n                  'thereanyone': 'there anyone', 'electronegativty': 'electronegativity',\n                  'commissiioned': 'commissioned', 'earvphone': 'earphone', 'condtioners': 'conditioners',\n                  'demonetistaion': 'demonetization', 'ballonets': 'ballo nets', 'DoneClaim': 'Done Claim',\n                  'alimoney': 'alimony', 'iodopovidone': 'iodo povidone', 'bonesetters': 'bone setters',\n                  'componendo': 'compon endo', 'probationees': 'probationers', 'one300': 'one 300',\n                  'nonelectrolyte': 'non electrolyte', 'ozonedepletion': 'ozone depletion',\n                  'Stonehart': 'Stone hart', 'Vodafone2': 'Vodafones', 'chaparone': 'chaperone',\n                  'Noonein': 'Noo nein', 'Frosione': 'Erosion', 'IPhone7': 'Iphone', 'pentanone': 'penta none',\n                  'poneglyphs': 'pone glyphs', 'cyclohexenone': 'cyclohexanone', 'marlstone': 'marls tone',\n                  'androneda': 'andromeda', 'iphone8': 'iPhone', 'acidtone': 'acid tone',\n                  'noneconomically': 'non economically', 'Honeyfund': 'Honey fund', 'germanophone': 'Germanophobe',\n                  'Democratizationed': 'Democratization ed', 'haoneymoon': 'honeymoon', 'iPhone7': 'iPhone 7',\n                  'someonewith': 'some onewith', 'Hexanone': 'Hexa none', 'bonespur': 'bones pur',\n                  'sisterzoned': 'sister zoned', 'HasAnyone': 'Has Anyone',\n                  'stonepelters': 'stone pelters', 'Chronexia': 'Chronaxia', 'brotherzone': 'brother zone',\n                  'brotherzoned': 'brother zoned', 'fonecare': 'f onecare', 'nonexsistence': 'nonexistence',\n                  'conents': 'contents', 'phonecases': 'phone cases', 'Commissionerates': 'Commissioner ates',\n                  'activemoney': 'active money', 'dingtone': 'ding tone', 'wheatestone': 'wheatstone',\n                  'chiropractorone': 'chiropractor one', 'heeadphones': 'headphones', 'Maimonedes': 'Maimonides',\n                  'onepiecedeals': 'onepiece deals', 'oneblade': 'one blade', 'venetioned': 'Venetianed',\n                  'sunnyleone': 'sunny leone', 'prendisone': 'prednisone', 'Anglosaxophone': 'Anglo saxophone',\n                  'Blackphones': 'Black phones', 'jionee': 'jinnee', 'chromonema': 'chromo nema',\n                  'iodoketones': 'iodo ketones', 'demonetizations': 'demonetization', 'aomeone': 'someone',\n                  'trillonere': 'trillones', 'abandonee': 'abandon',\n                  'MasterColonel': 'Master Colonel', 'fronend': 'friend', 'Wildstone': 'Wilds tone',\n                  'patitioned': 'petitioned', 'lonewolfs': 'lone wolfs', 'Spectrastone': 'Spectra stone',\n                  'dishonerable': 'dishonorable', 'poisiones': 'poisons',\n                  'condioner': 'conditioner', 'unpermissioned': 'unper missioned', 'friedzone': 'fried zone',\n                  'umumoney': 'umu money', 'anyonestudied': 'anyone studied', 'dictioneries': 'dictionaries',\n                  'nosebone': 'nose bone', 'ofVodafone': 'of Vodafone',\n                  'Yumstone': 'Yum stone', 'oxandrolonesteroid': 'oxandrolone steroid',\n                  'Mifeprostone': 'Mifepristone', 'pheramones': 'pheromones',\n                  'sinophone': 'Sinophobe', 'peloponesian': 'peloponnesian', 'michrophone': 'microphone',\n                  'commissionets': 'commissioners', 'methedone': 'methadone', 'cobditioners': 'conditioners',\n                  'urotone': 'protone', 'smarthpone': 'smartphone', 'conecTU': 'connect you', 'beloney': 'boloney',\n                  'comfortzone': 'comfort zone', 'testostersone': 'testosterone', 'camponente': 'component',\n                  'Idonesia': 'Indonesia', 'dolostones': 'dolostone', 'psiphone': 'psi phone',\n                  'ceftriazone': 'ceftriaxone', 'feelonely': 'feel onely', 'monetation': 'moderation',\n                  'activationenergy': 'activation energy', 'moneydriven': 'money driven',\n                  'staionery': 'stationery', 'zoneflex': 'zone flex', 'moneycash': 'money cash',\n                  'conectiin': 'connection', 'Wannaone': 'Wanna one',\n                  'Pictones': 'Pict ones', 'demonentization': 'demonetization',\n                  'phenonenon': 'phenomenon', 'evenafter': 'even after', 'Sevenfriday': 'Seven friday',\n                  'Devendale': 'Evendale', 'theeventchronicle': 'the event chronicle',\n                  'seventysomething': 'seventy something', 'sevenpointed': 'seven pointed',\n                  'richfeel': 'rich feel', 'overfeel': 'over feel', 'feelingstupid': 'feeling stupid',\n                  'Photofeeler': 'Photo feeler', 'feelomgs': 'feelings', 'feelinfs': 'feelings',\n                  'PlayerUnknown': 'Player Unknown', 'Playerunknown': 'Player unknown', 'knowlefge': 'knowledge',\n                  'knowledgd': 'knowledge', 'knowledeg': 'knowledge', 'knowble': 'Knowle', 'Howknow': 'Howk now',\n                  'knowledgeWoods': 'knowledge Woods', 'knownprogramming': 'known programming',\n                  'selfknowledge': 'self knowledge', 'knowldage': 'knowledge', 'knowyouve': 'know youve',\n                  'aknowlege': 'knowledge', 'Audetteknown': 'Audette known', 'knowlegdeable': 'knowledgeable',\n                  'trueoutside': 'true outside', 'saynthesize': 'synthesize', 'EssayTyper': 'Essay Typer',\n                  'meesaya': 'mee saya', 'Rasayanam': 'Rasayan am', 'fanessay': 'fan essay', 'momsays': 'moms ays',\n                  'sayying': 'saying', 'saydaw': 'say daw', 'Fanessay': 'Fan essay', 'theyreally': 'they really',\n                  'gayifying': 'gayed up with homosexual love', 'gayke': 'gay Online retailers',\n                  'Lingayatism': 'Lingayat',\n                  'macapugay': 'Macaulay', 'jewsplain': 'jews plain',\n                  'banggood': 'bang good', 'goodfriends': 'good friends',\n                  'goodfirms': 'good firms', 'Banggood': 'Bang good', 'dogooder': 'do gooder',\n                  'stillshots': 'stills hots', 'stillsuits': 'still suits', 'panromantic': 'pan romantic',\n                  'paracommando': 'para commando', 'romantize': 'romanize', 'manupulative': 'manipulative',\n                  'manjha': 'mania', 'mankrit': 'mank rit',\n                  'heteroromantic': 'hetero romantic', 'pulmanery': 'pulmonary', 'manpads': 'man pads',\n                  'supermaneuverable': 'super maneuverable', 'mandatkry': 'mandatory', 'armanents': 'armaments',\n                  'manipative': 'mancipative', 'himanity': 'humanity', 'maneuever': 'maneuver',\n                  'Kumarmangalam': 'Kumar mangalam', 'Brahmanwadi': 'Brahman wadi',\n                  'exserviceman': 'ex serviceman',\n                  'managewp': 'managed', 'manies': 'many', 'recordermans': 'recorder mans',\n                  'Feymann': 'Heymann', 'salemmango': 'salem mango', 'manufraturing': 'manufacturing',\n                  'sreeman': 'freeman', 'tamanaa': 'Tamanac', 'chlamydomanas': 'chlamydomonas',\n                  'comandant': 'commandant', 'huemanity': 'humanity', 'manaagerial': 'managerial',\n                  'lithromantics': 'lith romantics',\n                  'geramans': 'germans', 'Nagamandala': 'Naga mandala', 'humanitariarism': 'humanitarianism',\n                  'wattman': 'watt man', 'salesmanago': 'salesman ago', 'Washwoman': 'Wash woman',\n                  'rammandir': 'ram mandir', 'nomanclature': 'nomenclature', 'Haufman': 'Kaufman',\n                  'prefomance': 'performance', 'ramanunjan': 'Ramanujan', 'Freemansonry': 'Freemasonry',\n                  'supermaneuverability': 'super maneuverability', 'manstruate': 'menstruate',\n                  'Tarumanagara': 'Taruma nagara', 'RomanceTale': 'Romance Tale', 'heteromantic': 'hete romantic',\n                  'terimanals': 'terminals', 'womansplaining': 'wo mansplaining',\n                  'performancelearning': 'performance learning', 'sociomantic': 'sciomantic',\n                  'batmanvoice': 'batman voice', 'PerformanceTesting': 'Performance Testing',\n                  'manorialism': 'manorial ism', 'newscommando': 'news commando',\n                  'Entwicklungsroman': 'Entwicklungs roman',\n                  'Kunstlerroman': 'Kunstler roman', 'bodhidharman': 'Bodhidharma', 'Howmaney': 'How maney',\n                  'manufucturing': 'manufacturing', 'remmaning': 'remaining', 'rangeman': 'range man',\n                  'mythomaniac': 'mythomania', 'katgmandu': 'katmandu',\n                  'Superowoman': 'Superwoman', 'Rahmanland': 'Rahman land', 'Dormmanu': 'Dormant',\n                  'Geftman': 'Gentman', 'manufacturig': 'manufacturing', 'bramanistic': 'Brahmanistic',\n                  'padmanabhanagar': 'padmanabhan agar', 'homoromantic': 'homo romantic', 'femanists': 'feminists',\n                  'demihuman': 'demi human', 'manrega': 'Manresa', 'Pasmanda': 'Pas manda',\n                  'manufacctured': 'manufactured', 'remaninder': 'remainder', 'Marimanga': 'Mari manga',\n                  'Sloatman': 'Sloat man', 'manlet': 'man let', 'perfoemance': 'performance',\n                  'mangolian': 'mongolian', 'mangekyu': 'mange kyu', 'mansatory': 'mandatory',\n                  'managemebt': 'management', 'manufctures': 'manufactures', 'Bramanical': 'Brahmanical',\n                  'manaufacturing': 'manufacturing', 'Lakhsman': 'Lakhs man', 'Sarumans': 'Sarum ans',\n                  'mangalasutra': 'mangalsutra', 'Germanised': 'German ised',\n                  'managersworking': 'managers working', 'cammando': 'commando', 'mandrillaris': 'mandrill aris',\n                  'Emmanvel': 'Emmarvel', 'manupalation': 'manipulation', 'welcomeromanian': 'welcome romanian',\n                  'humanfemale': 'human female', 'mankirt': 'mankind', 'Haffmann': 'Hoffmann',\n                  'Panromantic': 'Pan romantic', 'demantion': 'detention', 'Suparwoman': 'Superwoman',\n                  'parasuramans': 'parasuram ans', 'sulmann': 'Suilmann', 'Shubman': 'Subman',\n                  'manspread': 'man spread', 'mandingan': 'Mandingan', 'mandalikalu': 'mandalika lu',\n                  'manufraturer': 'manufacturer', 'Wedgieman': 'Wedgie man', 'manwues': 'manages',\n                  'humanzees': 'human zees', 'Steymann': 'Stedmann', 'Jobberman': 'Jobber man',\n                  'maniquins': 'mani quins', 'biromantical': 'bi romantical', 'Rovman': 'Roman',\n                  'pyromantic': 'pyro mantic', 'Tastaman': 'Rastaman', 'Spoolman': 'Spool man',\n                  'Subramaniyan': 'Subramani yan', 'abhimana': 'abhiman a', 'manholding': 'man holding',\n                  'seviceman': 'serviceman', 'womansplained': 'womans plained', 'manniya': 'mania',\n                  'Bhraman': 'Braman', 'Laakman': 'Layman', 'mansturbate': 'masturbate',\n                  'Sulamaniya': 'Sulamani ya', 'demanters': 'decanters', 'postmanare': 'postman are',\n                  'mannualy': 'annual', 'rstman': 'Rotman', 'permanentjobs': 'permanent jobs',\n                  'Allmang': 'All mang', 'TradeCommander': 'Trade Commander', 'BasedStickman': 'Based Stickman',\n                  'Deshabhimani': 'Desha bhimani', 'manslamming': 'mans lamming', 'Brahmanwad': 'Brahman wad',\n                  'fundemantally': 'fundamentally', 'supplemantary': 'supplementary', 'egomanias': 'ego manias',\n                  'manvantar': 'Manvantara', 'spymania': 'spy mania', 'mangonada': 'mango nada',\n                  'manthras': 'mantras', 'Humanpark': 'Human park', 'manhuas': 'mahuas',\n                  'manterrupting': 'interrupting', 'dermatillomaniac': 'dermatillomania',\n                  'performancies': 'performances', 'manipulant': 'manipulate',\n                  'painterman': 'painter man', 'mangalik': 'manglik',\n                  'Neurosemantics': 'Neuro semantics', 'discrimantion': 'discrimination',\n                  'Womansplaining': 'feminist', 'mongodump': 'mongo dump', 'roadgods': 'road gods',\n                  'Oligodendraglioma': 'Oligodendroglioma', 'unrightly': 'un rightly', 'Janewright': 'Jane wright',\n                  ' righten ': ' tighten ', 'brightiest': 'brightest',\n                  'frighter': 'fighter', 'righteouness': 'righteousness', 'triangleright': 'triangle right',\n                  'Brightspace': 'Brights pace', 'techinacal': 'technical', 'chinawares': 'china wares',\n                  'Vancouever': 'Vancouver', 'cheverlet': 'cheveret', 'deverstion': 'diversion',\n                  'everbodys': 'everybody', 'Dramafever': 'Drama fever', 'reverificaton': 'reverification',\n                  'canterlever': 'canter lever', 'keywordseverywhere': 'keywords everywhere',\n                  'neverunlearned': 'never unlearned', 'everyfirst': 'every first',\n                  'neverhteless': 'nevertheless', 'clevercoyote': 'clever coyote', 'irrevershible': 'irreversible',\n                  'achievership': 'achievers hip', 'easedeverything': 'eased everything', 'youbever': 'you bever',\n                  'everperson': 'ever person', 'everydsy': 'everyday', 'whemever': 'whenever',\n                  'everyonr': 'everyone', 'severiity': 'severity', 'narracist': 'nar racist',\n                  'racistly': 'racist', 'takesuch': 'take such', 'mystakenly': 'mistakenly',\n                  'shouldntake': 'shouldnt take', 'Kalitake': 'Kali take', 'msitake': 'mistake',\n                  'straitstimes': 'straits times', 'timefram': 'timeframe', 'watchtime': 'watch time',\n                  'timetraveling': 'timet raveling', 'peactime': 'peacetime', 'timetabe': 'timetable',\n                  'cooktime': 'cook time', 'blocktime': 'block time', 'timesjobs': 'times jobs',\n                  'timesence': 'times ence', 'Touchtime': 'Touch time', 'timeloop': 'time loop',\n                  'subcentimeter': 'sub centimeter', 'timejobs': 'time jobs', 'Guardtime': 'Guard time',\n                  'realtimepolitics': 'realtime politics', 'loadingtimes': 'loading times',\n                  'timesnow': '24-hour English news channel in India', 'timesspark': 'times spark',\n                  'timetravelling': 'timet ravelling',\n                  'antimeter': 'anti meter', 'timewaste': 'time waste', 'cryptochristians': 'crypto christians',\n                  'Whatcould': 'What could', 'becomesdouble': 'becomes double', 'deathbecomes': 'death becomes',\n                  'youbecome': 'you become', 'greenseer': 'people who possess the magical ability',\n                  'rseearch': 'research', 'homeseek': 'home seek',\n                  'Greenseer': 'people who possess the magical ability', 'starseeders': 'star seeders',\n                  'seekingmillionaire': 'seeking millionaire', 'see\\u202c': 'see',\n                  'seeies': 'series', 'CodeAgon': 'Code Agon',\n                  'royago': 'royal', 'Dragonkeeper': 'Dragon keeper', 'mcgreggor': 'McGregor',\n                  'catrgory': 'category', 'Dragonknight': 'Dragon knight', 'Antergos': 'Anteros',\n                  'togofogo': 'togo fogo', 'mongorestore': 'mongo restore', 'gorgops': 'gorgons',\n                  'withgoogle': 'with google', 'goundar': 'Gondar', 'algorthmic': 'algorithmic',\n                  'goatnuts': 'goat nuts', 'vitilgo': 'vitiligo', 'polygony': 'poly gony',\n                  'digonals': 'diagonals', 'Luxemgourg': 'Luxembourg', 'UCSanDiego': 'UC SanDiego',\n                  'Ringostat': 'Ringo stat', 'takingoff': 'taking off', 'MongoImport': 'Mongo Import',\n                  'alggorithms': 'algorithms', 'dragonknight': 'dragon knight', 'negotiatior': 'negotiation',\n                  'gomovies': 'go movies', 'Withgott': 'Without',\n                  'categoried': 'categories', 'Stocklogos': 'Stock logos', 'Pedogogical': 'Pedological',\n                  'Wedugo': 'Wedge', 'golddig': 'gold dig', 'goldengroup': 'golden group',\n                  'merrigo': 'merligo', 'googlemapsAPI': 'googlemaps API', 'goldmedal': 'gold medal',\n                  'golemized': 'polemized', 'Caligornia': 'California', 'unergonomic': 'un ergonomic',\n                  'fAegon': 'wagon', 'vertigos': 'vertigo s', 'trigonomatry': 'trigonometry',\n                  'hypogonadic': 'hypogonadia', 'Mogolia': 'Mongolia', 'governmaent': 'government',\n                  'ergotherapy': 'ergo therapy', 'Bogosort': 'Bogo sort', 'goalwise': 'goal wise',\n                  'alogorithms': 'algorithms', 'MercadoPago': 'Mercado Pago', 'rivigo': 'rivi go',\n                  'govshutdown': 'gov shutdown', 'gorlfriend': 'girlfriend',\n                  'stategovt': 'state govt', 'Chickengonia': 'Chicken gonia', 'Yegorovich': 'Yegorov ich',\n                  'regognitions': 'recognitions', 'gorichen': 'Gori Chen Mountain',\n                  'goegraphies': 'geographies', 'gothras': 'goth ras', 'belagola': 'bela gola',\n                  'snapragon': 'snapdragon', 'oogonial': 'oogonia l', 'Amigofoods': 'Amigo foods',\n                  'Sigorn': 'son of Styr', 'algorithimic': 'algorithmic',\n                  'innermongolians': 'inner mongolians', 'ArangoDB': 'Arango DB', 'zigolo': 'gigolo',\n                  'regognized': 'recognized', 'Moongot': 'Moong ot', 'goldquest': 'gold quest',\n                  'catagorey': 'category', 'got7': 'got', 'jetbingo': 'jet bingo', 'Dragonchain': 'Dragon chain',\n                  'catwgorized': 'categorized', 'gogoro': 'gogo ro', 'Tobagoans': 'Tobago ans',\n                  'digonal': 'di gonal', 'algoritmic': 'algorismic', 'dragonflag': 'dragon flag',\n                  'Indigoflight': 'Indigo flight',\n                  'governening': 'governing', 'ergosphere': 'ergo sphere',\n                  'pingo5': 'pingo', 'Montogo': 'montego', 'Rivigo': 'technology-enabled logistics company',\n                  'Jigolo': 'Gigolo', 'phythagoras': 'pythagoras', 'Mangolian': 'Mongolian',\n                  'forgottenfaster': 'forgotten faster', 'stargold': 'a Hindi movie channel',\n                  'googolplexain': 'googolplexian', 'corpgov': 'corp gov',\n                  'govtribe': 'provides real-time federal contracting market intel',\n                  'dragonglass': 'dragon glass', 'gorakpur': 'Gorakhpur', 'MangoPay': 'Mango Pay',\n                  'chigoe': 'sub-tropical climates', 'BingoBox': 'an investment company', '走go': 'go',\n                  'followingorder': 'following order', 'pangolinminer': 'pangolin miner',\n                  'negosiation': 'negotiation', 'lexigographers': 'lexicographers', 'algorithom': 'algorithm',\n                  'unforgottable': 'unforgettable', 'wellsfargoemail': 'wellsfargo email',\n                  'daigonal': 'diagonal', 'Pangoro': 'cantankerous Pokemon', 'negotiotions': 'negotiations',\n                  'Swissgolden': 'Swiss golden', 'google4': 'google', 'Agoraki': 'Ago raki',\n                  'Garthago': 'Carthago', 'Stegosauri': 'stegosaurus', 'ergophobia': 'ergo phobia',\n                  'bigolive': 'big olive', 'bittergoat': 'bitter goat', 'naggots': 'faggots',\n                  'googology': 'online encyclopedia', 'algortihms': 'algorithms', 'bengolis': 'Bengalis',\n                  'fingols': 'Finnish people are supposedly descended from Mongols',\n                  'savethechildren': 'save thechildren',\n                  'stopings': 'stoping', 'stopsits': 'stop sits', 'stopsigns': 'stop signs',\n                  'Galastop': 'Galas top', 'pokestops': 'pokes tops', 'forcestop': 'forces top',\n                  'Hopstop': 'Hops top', 'stoppingexercises': 'stopping exercises', 'coinstop': 'coins top',\n                  'stoppef': 'stopped', 'workaway': 'work away', 'snazzyway': 'snazzy way',\n                  'Rewardingways': 'Rewarding ways', 'cloudways': 'cloud ways', 'Cloudways': 'Cloud ways',\n                  'Brainsway': 'Brains way', 'nesraway': 'nearaway',\n                  'AlwaysHired': 'Always Hired', 'expessway': 'expressway', 'Syncway': 'Sync way',\n                  'LeewayHertz': 'Blockchain Company', 'towayrds': 'towards', 'swayable': 'sway able',\n                  'Telloway': 'Tello way', 'palsmodium': 'plasmodium', 'Gobackmodi': 'Goback modi',\n                  'comodies': 'corodies', 'islamphobic': 'islam phobic', 'islamphobia': 'islam phobia',\n                  'citiesbetter': 'cities better', 'betterv3': 'better', 'betterDtu': 'better Dtu',\n                  'Babadook': 'a horror drama film', 'Ahemadabad': 'Ahmadabad', 'faidabad': 'Faizabad',\n                  'Amedabad': 'Ahmedabad', 'kabadii': 'kabaddi', 'badmothing': 'badmouthing',\n                  'badminaton': 'badminton', 'badtameezdil': 'badtameez dil', 'badeffects': 'bad effects',\n                  '∠bad': 'bad', 'ahemadabad': 'Ahmadabad', 'embaded': 'embased', 'Isdhanbad': 'Is dhanbad',\n                  'badgermoles': 'enormous, blind mammal', 'allhabad': 'Allahabad', 'ghazibad': 'ghazi bad',\n                  'htderabad': 'Hyderabad', 'Auragabad': 'Aurangabad', 'ahmedbad': 'Ahmedabad',\n                  'ahmdabad': 'Ahmadabad', 'alahabad': 'Allahabad',\n                  'Hydeabad': 'Hyderabad', 'Gyroglove': 'wearable technology', 'foodlovee': 'food lovee',\n                  'slovenised': 'slovenia', 'handgloves': 'hand gloves', 'lovestep': 'love step',\n                  'lovejihad': 'love jihad', 'RolloverBox': 'Rollover Box', 'stupidedt': 'stupidest',\n                  'toostupid': 'too stupid',\n                  'pakistanisbeautiful': 'pakistanis beautiful', 'ispakistan': 'is pakistan',\n                  'inpersonations': 'impersonations', 'medicalperson': 'medical person',\n                  'interpersonation': 'inter personation', 'workperson': 'work person',\n                  'personlich': 'person lich', 'persoenlich': 'person lich',\n                  'middleperson': 'middle person', 'personslized': 'personalized',\n                  'personifaction': 'personification', 'welcomemarriage': 'welcome marriage',\n                  'come2': 'come to', 'upcomedians': 'up comedians', 'overvcome': 'overcome',\n                  'talecome': 'tale come', 'cometitive': 'competitive', 'arencome': 'aren come',\n                  'achecomes': 'ache comes', '」come': 'come',\n                  'comepleted': 'completed', 'overcomeanxieties': 'overcome anxieties',\n                  'demigirl': 'demi girl', 'gridgirl': 'female models of the race', 'halfgirlfriend': 'half girlfriend',\n                  'girlriend': 'girlfriend', 'fitgirl': 'fit girl', 'girlfrnd': 'girlfriend', 'awrong': 'aw rong',\n                  'northcap': 'north cap', 'productionsupport': 'production support',\n                  'Designbold': 'Online Photo Editor Design Studio',\n                  'skyhold': 'sky hold', 'shuoldnt': 'shouldnt', 'anarold': 'Android', 'yaerold': 'year old',\n                  'soldiders': 'soldiers', 'indrold': 'Android', 'blindfoldedly': 'blindfolded',\n                  'overcold': 'over cold', 'Goldmont': 'microarchitecture in Intel', 'boldspot': 'bolds pot',\n                  'Rankholders': 'Rank holders', 'cooldrink': 'cool drink', 'beltholders': 'belt holders',\n                  'GoldenDict': 'open-source dictionary program', 'softskill': 'softs kill',\n                  'Cooldige': 'the 30th president of the United States',\n                  'newkiller': 'new killer', 'skillselect': 'skills elect', 'nonskilled': 'non skilled',\n                  'killyou': 'kill you', 'Skillport': 'Army e-Learning Program', 'unkilled': 'un killed',\n                  'killikng': 'killing', 'killograms': 'kilograms',\n                  'Worldkillers': 'World killers', 'reskilled': 'skilled',\n                  'killedshivaji': 'killed shivaji', 'honorkillings': 'honor killings',\n                  'skillclasses': 'skill classes', 'microskills': 'micros kills',\n                  'Skillselect': 'Skills elect', 'ratkill': 'rat kill',\n                  'pleasegive': 'please give', 'flashgive': 'flash give',\n                  'southerntelescope': 'southern telescope', 'westsouth': 'west south',\n                  'southAfricans': 'south Africans', 'Joboutlooks': 'Job outlooks', 'joboutlook': 'job outlook',\n                  'Outlook365': 'Outlook 365', 'Neulife': 'Neu life', 'qualifeid': 'qualified',\n                  'nullifed': 'nullified', 'lifeaffect': 'life affect', 'lifestly': 'lifestyle',\n                  'aristocracylifestyle': 'aristocracy lifestyle', 'antilife': 'anti life',\n                  'afterafterlife': 'after afterlife', 'lifestylye': 'lifestyle', 'prelife': 'pre life',\n                  'lifeute': 'life ute', 'liferature': 'literature',\n                  'securedlife': 'secured life', 'doublelife': 'double life', 'antireligion': 'anti religion',\n                  'coreligionist': 'co religionist', 'petrostates': 'petro states', 'otherstates': 'others tates',\n                  'spacewithout': 'space without', 'withoutyou': 'without you',\n                  'withoutregistered': 'without registered', 'weightwithout': 'weight without',\n                  'withoutcheck': 'without check', 'milkwithout': 'milk without',\n                  'Highschoold': 'High school', 'memoney': 'money', 'moneyof': 'mony of', 'Oneplus': 'OnePlus',\n                  'OnePlus': 'Chinese smartphone manufacturer', 'Beerus': 'the God of Destruction',\n                  'takeoverr': 'takeover', 'demonetizedd': 'demonetized', 'polyhouse': 'Polytunnel',\n                  'Elitmus': 'eLitmus', 'eLitmus': 'Indian company that helps companies in hiring employees',\n                  'becone': 'become', 'nestaway': 'nest away', 'takeoverrs': 'takeovers', 'Istop': 'I stop',\n                  'Austira': 'Australia', 'germeny': 'Germany', 'mansoon': 'man soon',\n                  'worldmax': 'wholesaler of drum parts',\n                  'ammusement': 'amusement', 'manyare': 'many are', 'supplymentary': 'supply mentary',\n                  'timesup': 'times up', 'homologus': 'homologous', 'uimovement': 'ui movement', 'spause': 'spouse',\n                  'aesexual': 'asexual', 'Iovercome': 'I overcome', 'developmeny': 'development',\n                  'hindusm': 'hinduism', 'sexpat': 'sex tourism', 'sunstop': 'sun stop', 'polyhouses': 'Polytunnel',\n                  'usefl': 'useful', 'Fundamantal': 'fundamental', 'environmentai': 'environmental',\n                  'Redmi': 'Xiaomi Mobile', 'Loy Machedo': ' Motivational Speaker ', 'unacademy': 'Unacademy',\n                  'Boruto': 'Naruto Next Generations', 'Upwork': 'Up work',\n                  'Unacademy': 'educational technology company',\n                  'HackerRank': 'Hacker Rank', 'upwork': 'up work', 'Chromecast': 'Chrome cast',\n                  'microservices': 'micro services', 'Undertale': 'video game', 'undergraduation': 'under graduation',\n                  'chapterwise': 'chapter wise', 'twinflame': 'twin flame', 'Hotstar': 'Hot star',\n                  'blockchains': 'blockchain',\n                  'darkweb': 'dark web', 'Microservices': 'Micro services', 'Nearbuy': 'Nearby',\n                  ' Padmaavat ': ' Padmavati ', ' padmavat ': ' Padmavati ', ' Padmaavati ': ' Padmavati ',\n                  ' Padmavat ': ' Padmavati ', ' internshala ': ' internship and online training platform in India ',\n                  'dream11': ' fantasy sports platform in India ', 'conciousnesss': 'consciousnesses',\n                  'Dream11': ' fantasy sports platform in India ', 'cointry': 'country', ' coinvest ': ' invest ',\n                  '23 andme': 'privately held personal genomics and biotechnology company in California',\n                  'Trumpism': 'philosophy and politics espoused by Donald Trump',\n                  'Trumpian': 'viewpoints of President Donald Trump', 'Trumpists': 'admirer of Donald Trump',\n                  'coincidents': 'coincidence', 'coinsized': 'coin sized', 'coincedences': 'coincidences',\n                  'cointries': 'countries', 'coinsidered': 'considered', 'coinfirm': 'confirm',\n                  'humilates':'humiliates', 'vicevice':'vice vice', 'politicak':'political', 'Sumaterans':'Sumatrans',\n                  'Kamikazis':'Kamikazes', 'unmoraled':'unmoral', 'eduacated':'educated', 'moraled':'morale',\n                  'Amharc':'Amarc', 'where Burkhas':'wear Burqas', 'Baloochistan':'Balochistan', 'durgahs':'durgans',\n                  'illigitmate':'illegitimate', 'hillum':'helium','treatens':'threatens','mutiliating':'mutilating',\n                  'speakingly':'speaking', 'pretex':'pretext', 'menstruateion':'menstruation', \n                  'genocidizing':'genociding', 'maratis':'Maratism','Parkistinian':'Pakistani', 'SPEICIAL':'SPECIAL',\n                  'REFERNECE':'REFERENCE', 'provocates':'provokes', 'FAMINAZIS':'FEMINAZIS', 'repugicans':'republicans',\n                  'tonogenesis':'tone', 'winor':'win', 'redicules':'ridiculous', 'Beluchistan':'Balochistan', \n                  'volime':'volume', 'namaj':'namaz', 'CONgressi':'Congress', 'Ashifa':'Asifa', 'queffing':'queefing',\n                  'montheistic':'nontheistic', 'Rajsthan':'Rajasthan', 'Rajsthanis':'Rajasthanis', 'specrum':'spectrum',\n                  'brophytes':'bryophytes', 'adhaar':'Adhara', 'slogun':'slogan', 'harassd':'harassed',\n                  'transness':'trans gender', 'Insdians':'Indians', 'Trampaphobia':'Trump aphobia', 'attrected':'attracted',\n                  'Yahtzees':'Yahtzee', 'thiests':'atheists', 'thrir':'their', 'extraterestrial':'extraterrestrial',\n                  'silghtest':'slightest', 'primarty':'primary','brlieve':'believe', 'fondels':'fondles',\n                  'loundly':'loudly', 'bootythongs':'booty thongs', 'understamding':'understanding', 'degenarate':'degenerate',\n                  'narsistic':'narcistic', 'innerskin':'inner skin','spectulated':'speculated', 'hippocratical':'Hippocratical',\n                  'itstead':'instead', 'parralels':'parallels', 'sloppers':'slippers'\n                  }\n\ndef clean_bad_case_words(text):\n    for bad_word in bad_case_words:\n        if bad_word in text:\n            text = text.replace(bad_word, bad_case_words[bad_word])\n    return text\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"057be0dae264abe38d84f88f55faef0d5b6ad1dc"},"cell_type":"code","source":"mis_connect_list = ['(W|w)hat', '(W|w)hy', '(H|h)ow', '(W|w)hich', '(W|w)here', '(W|w)ill']\nmis_connect_re = re.compile('(%s)' % '|'.join(mis_connect_list))\n\nmis_spell_mapping = {'whattsup': 'WhatsApp', 'whatasapp':'WhatsApp', 'whatsupp':'WhatsApp', \n                      'whatcus':'what cause', 'arewhatsapp': 'are WhatsApp', 'Hwhat':'what',\n                      'Whwhat': 'What', 'whatshapp':'WhatsApp', 'howhat':'how that',\n                      # why\n                      'Whybis':'Why is', 'laowhy86':'Foreigners who do not respect China',\n                      'Whyco-education':'Why co-education',\n                      # How\n                      \"Howddo\":\"How do\", 'Howeber':'However', 'Showh':'Show',\n                      \"Willowmagic\":'Willow magic', 'WillsEye':'Will Eye', 'Williby':'will by'}\ndef spacing_some_connect_words(text):\n    \"\"\"\n    'Whyare' -> 'Why are'\n    \"\"\"\n    ori = text\n    for error in mis_spell_mapping:\n        if error in text:\n            text = text.replace(error, mis_spell_mapping[error])\n            \n    # what\n    text = re.sub(r\" (W|w)hat+(s)*[A|a]*(p)+ \", \" WhatsApp \", text)\n    text = re.sub(r\" (W|w)hat\\S \", \" What \", text)\n    text = re.sub(r\" \\S(W|w)hat \", \" What \", text)\n    # why\n    text = re.sub(r\" (W|w)hy\\S \", \" Why \", text)\n    text = re.sub(r\" \\S(W|w)hy \", \" Why \", text)\n    # How\n    text = re.sub(r\" (H|h)ow\\S \", \" How \", text)\n    text = re.sub(r\" \\S(H|h)ow \", \" How \", text)\n    # which\n    text = re.sub(r\" (W|w)hich\\S \", \" Which \", text)\n    text = re.sub(r\" \\S(W|w)hich \", \" Which \", text)\n    # where\n    text = re.sub(r\" (W|w)here\\S \", \" Where \", text)\n    text = re.sub(r\" \\S(W|w)here \", \" Where \", text)\n    # \n    text = mis_connect_re.sub(r\" \\1 \", text)\n    text = text.replace(\"What sApp\", 'WhatsApp')\n    \n    text = remove_space(text)\n    return text\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5bec9f6fb52d8e508e943d4f9645623eadba0dfe"},"cell_type":"code","source":"# clean repeated letters\ndef clean_repeat_words(text):\n    text = text.replace(\"img\", \"ing\")\n\n    text = re.sub(r\"(I|i)(I|i)+ng\", \"ing\", text)\n    text = re.sub(r\"(L|l)(L|l)(L|l)+y\", \"lly\", text)\n    text = re.sub(r\"(A|a)(A|a)(A|a)+\", \"a\", text)\n    text = re.sub(r\"(C|c)(C|c)(C|c)+\", \"cc\", text)\n    text = re.sub(r\"(D|d)(D|d)(D|d)+\", \"dd\", text)\n    text = re.sub(r\"(E|e)(E|e)(E|e)+\", \"ee\", text)\n    text = re.sub(r\"(F|f)(F|f)(F|f)+\", \"ff\", text)\n    text = re.sub(r\"(G|g)(G|g)(G|g)+\", \"gg\", text)\n    text = re.sub(r\"(I|i)(I|i)(I|i)+\", \"i\", text)\n    text = re.sub(r\"(K|k)(K|k)(K|k)+\", \"k\", text)\n    text = re.sub(r\"(L|l)(L|l)(L|l)+\", \"ll\", text)\n    text = re.sub(r\"(M|m)(M|m)(M|m)+\", \"mm\", text)\n    text = re.sub(r\"(N|n)(N|n)(N|n)+\", \"nn\", text)\n    text = re.sub(r\"(O|o)(O|o)(O|o)+\", \"oo\", text)\n    text = re.sub(r\"(P|p)(P|p)(P|p)+\", \"pp\", text)\n    text = re.sub(r\"(Q|q)(Q|q)+\", \"q\", text)\n    text = re.sub(r\"(R|r)(R|r)(R|r)+\", \"rr\", text)\n    text = re.sub(r\"(S|s)(S|s)(S|s)+\", \"ss\", text)\n    text = re.sub(r\"(T|t)(T|t)(T|t)+\", \"tt\", text)\n    text = re.sub(r\"(V|v)(V|v)+\", \"v\", text)\n    text = re.sub(r\"(Y|y)(Y|y)(Y|y)+\", \"y\", text)\n    text = re.sub(r\"plzz+\", \"please\", text)\n    text = re.sub(r\"(Z|z)(Z|z)(Z|z)+\", \"zz\", text)\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84e288214a41a3ce7d22e4e48adadda2af1982c7"},"cell_type":"code","source":"def preprocess(text):\n    \"\"\"\n    preprocess text main steps\n    \"\"\"\n    text = remove_space(text)\n    text = clean_special_punctuations(text)\n    text = clean_number(text)\n    text = pre_clean_rare_words(text)\n    text = decontracted(text)\n    text = clean_latex(text)\n    text = clean_misspell(text)\n    text = spacing_punctuation(text)\n    text = spacing_some_connect_words(text)\n    text = clean_bad_case_words(text)\n    text = clean_repeat_words(text)\n    text = remove_space(text)\n    return text\n\ndef text_clean_wrapper(df):\n    df[\"question_text\"] = df[\"question_text\"].apply(preprocess)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9aa08d4147c80951b0f86450f71b6d342a59555"},"cell_type":"code","source":"%%time\ntrain_df = df_parallelize_run(train_df, text_clean_wrapper)\ntest_df = df_parallelize_run(test_df, text_clean_wrapper)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec6d452f78979d46866bc545c852f9b63526fac1"},"cell_type":"code","source":"%%time\noov_glove, oov_paragram, oov_fasttext, oov_google = vocab_check_coverage(train_df, test_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d3cf3a1b159c5dd8b56db6f469bdc63b9baff0d"},"cell_type":"markdown","source":"After cleaning the text the word coverage increased dramaticly.\n\n| Embedding  | Original | Text Cleaning |\n|----------|-------|------------|\n|Glove vocab founded| 33.92%| 72.59% |\n|Glove vocab founded in text| 88.20%| 99.44% |\n|Paragram vocab founded| 34.08%| 72.87% |\n|Paragram vocab founded in text| 88.21%| 99.45% |\n|FastText vocab founded| 31.63%| 68.32% |\n|FastText vocab founded in text| 87.74% | 99.38% |\n|Google vocab founded| 26.24%| 56.71% |\n|Google vocab founded in text| 87.26%| 88.01% |\n"},{"metadata":{"_uuid":"ae4f55dc634828ca98af2df818934171b30dfc3e"},"cell_type":"markdown","source":"Finally let's fixed the dash and point spacing bug!\n\n- spacing puncs except for '-'\n- get current vocabulary, and found the words that has '-'\n- check word that '-' in it whether has embedding\n- spacing '-'\n- fix these bug '-' words"},{"metadata":{"trusted":true,"_uuid":"1809e04c244cec7b6d95857b830afcca0535a8a9"},"cell_type":"code","source":"# get current vocabulary, and found the words that has '-'\ncur_vocabulary = set()\nfor text in tqdm(train_df['question_text'].values.tolist() + test_df['question_text'].values.tolist()):\n    words = text.split(' ')\n    cur_vocabulary.update(set(words))\n\nbug_punc_spacing_words_mapping = {}\nfor vocab in cur_vocabulary:\n    if '-' in vocab:\n        # whether the glove or para contain this word\n        if (vocab in embed_glove or vocab.capitalize() in embed_glove or vocab.lower() in embed_glove) and \\\n            (vocab in embed_paragram or vocab.lower() in embed_paragram):\n            bug_punc_spacing_words_mapping[f\" {' - '.join(vocab.split('-'))} \"] = f\" {vocab} \"\n    \n    if '.' in vocab:\n        if vocab.endswith('.'):\n            continue\n        \n        if (vocab in embed_glove or vocab.capitalize() in embed_glove or vocab.lower() in embed_glove) and \\\n            (vocab in embed_paragram or vocab.lower() in embed_paragram):\n            bug_punc_spacing_words_mapping[f\" {' . '.join(vocab.split('.'))} \"] = f\" {vocab} \"\n                                    \ndel bug_punc_spacing_words_mapping['  -  ']\nprint(f'found {len(bug_punc_spacing_words_mapping)} bug words')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"8125c0aa58e9c4e27ff46f64cd9a0f6bc409ea9a"},"cell_type":"code","source":"bug_punc_spacing_words_mapping","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08bdaa3bfa19772fccf978bc4ef1e58acad1e4f2"},"cell_type":"code","source":"def spacing_dash_point(text):\n    if '-' in text:\n        text = text.replace('-', ' - ')\n    if '.' in text:\n        text = text.replace('.', ' . ')\n    return text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"11f59a4a99b6033622d6ffc8b7e43a057d21540d"},"cell_type":"code","source":"%%time\ntrain_df[\"question_text\"] = train_df[\"question_text\"].apply(spacing_dash_point)\ntest_df[\"question_text\"] = test_df[\"question_text\"].apply(spacing_dash_point)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff8f3682016ed8e394ea0483beb05b3c6c2aa3e3"},"cell_type":"code","source":"def fix_dash_point_spacing_bug(text):\n    for bug_dash in bug_punc_spacing_words_mapping:\n        if bug_dash in text:\n            text = text.replace(bug_dash, bug_punc_spacing_words_mapping[bug_dash])\n    return text\n\ndef fix_dash_point_spacing_bug_wrapper(df):\n    df[\"question_text\"] = df[\"question_text\"].apply(fix_dash_point_spacing_bug)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"933781c1edd17ac23dc402103ce9cd1bb0efc56f"},"cell_type":"code","source":"%%time\ntrain_df = df_parallelize_run(train_df, fix_dash_point_spacing_bug_wrapper)\ntest_df = df_parallelize_run(test_df, fix_dash_point_spacing_bug_wrapper)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d8b2da7d05e67c14b4bebd83e5e2e752240ed5bb"},"cell_type":"code","source":"%%time\noov_glove, oov_paragram, oov_fasttext, oov_google = vocab_check_coverage(train_df, test_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"560cfb39373c7959df4cb20f3132175acbc5adec"},"cell_type":"code","source":"print('glove oov rate:', oov_glove['oov_rate'])\nprint('paragram oov rate:', oov_paragram['oov_rate'])\nprint('fasttext oov rate:', oov_fasttext['oov_rate'])\nprint('google oov rate:', oov_google['oov_rate'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c91571de8dfeb3ad5b53467fcf6b94636f937c57"},"cell_type":"markdown","source":"# Save cleaned datas"},{"metadata":{"trusted":true,"_uuid":"ecab9f09658001eecde90ea92c7751ea74bfef5a"},"cell_type":"code","source":"train_df.to_csv(\"cleaned_train.csv\", index=False)\ntest_df.to_csv(\"cleaned_test.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"defc342b0843173ddfc546368512b49c24395efc"},"cell_type":"markdown","source":"After fixing the dash and point spacing bug, the coverage continue improve a lot.\n\n| Embedding  | Original | Text Cleaning | Fix punc spacing bug  |\n|----------|-------|------------|------------------|\n|Glove vocab founded| 33.92%| 72.59% | 80.31%|\n|Glove vocab founded in text| 88.20%| 99.44% | 99.68%|\n|Paragram vocab founded| 34.08%| 72.87% | 80.52%|\n|Paragram vocab founded in text| 88.21%| 99.45% | 99.68%|\n|FastText vocab founded| 31.63%| 68.32% | 74.43%|\n|FastText vocab founded in text| 87.74% | 99.38% | 99.57%|\n|Google vocab founded| 26.24%| 56.71% | 64.71%|\n|Google vocab founded in text| 87.26%| 88.01% | 87.89%|\n"},{"metadata":{"trusted":true,"_uuid":"50eba5228bd7071f28ab7dc0ceb849335b9bbd83"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}