{"cells":[{"metadata":{"_cell_guid":"01959eac-42a4-9445-9d7f-06325684f193","_uuid":"15af3d81048d47b238782b5b998646d7ff69b9d4"},"cell_type":"markdown","source":"This comes from CPMP script in the Quora questions similarity challenge. With some modifications for this challenge.\n\n>> I have been struggling for a while on how to spell check questions while only using allowed data/software.  Here is the solution I am using now.\n>> It is an adaptation of Peter Norvig's spell checker.  It uses word2vec ordering of words to approximate word probabilities.  Indeed, Google word2vec apparently orders words >> in decreasing order of frequency in the training corpus."},{"metadata":{"_cell_guid":"1d7bd7dc-bae7-4313-856b-bdc10e447644","_uuid":"43d3228b7ea48b21012b901fe25a926b5096284f","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"36d0c3a7-e625-44d9-6f71-ae8df520a1cb","_uuid":"7dec3d822e9bc4c270d1b1929db9657ef2e14a0c","trusted":true},"cell_type":"code","source":"import gensim\nmodel = gensim.models.KeyedVectors.load_word2vec_format('../input/embeddings/GoogleNews-vectors-negative300/GoogleNews-vectors-negative300.bin', \n                                                        binary=True)\n\nwords = model.index2word\n\nw_rank = {}\nfor i,word in enumerate(words):\n    w_rank[word] = i\n\nWORDS = w_rank","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"e9756896-0a0a-81d0-66d9-821f7cdc8281","_uuid":"865b810a338431bd9771ca354872b0e7c9857d24"},"cell_type":"markdown","source":"The rest of the code is a simple modification of Peter Norvig's code. Instead of computing the frequency of each word we use the above rank."},{"metadata":{"_cell_guid":"c1a89822-cb6e-393d-ffaa-af0604ea68dc","_uuid":"a3eb42d3b562d7073524e14f25f8fa38b6bf6ede","trusted":true},"cell_type":"code","source":"import re\nfrom collections import Counter\n\ndef words(text): return re.findall(r'\\w+', text.lower())\n\ndef P(word): \n    \"Probability of `word`.\"\n    # use inverse of rank as proxy\n    # returns 0 if the word isn't in the dictionary\n    return - WORDS.get(word, 0)\n\ndef correction(word): \n    \"Most probable spelling correction for word.\"\n    return max(candidates(word), key=P)\n\ndef candidates(word): \n    \"Generate possible spelling corrections for word.\"\n    return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])\n\ndef known(words): \n    \"The subset of `words` that appear in the dictionary of WORDS.\"\n    return set(w for w in words if w in WORDS)\n\ndef edits1(word):\n    \"All edits that are one edit away from `word`.\"\n    letters    = 'abcdefghijklmnopqrstuvwxyz'\n    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]\n    deletes    = [L + R[1:]               for L, R in splits if R]\n    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]\n    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]\n    inserts    = [L + c + R               for L, R in splits for c in letters]\n    return set(deletes + transposes + replaces + inserts)\n\ndef edits2(word): \n    \"All edits that are two edits away from `word`.\"\n    return (e2 for e1 in edits1(word) for e2 in edits1(e1))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"26e763f3-e3fc-1415-0453-31cb390ab4a9","_uuid":"2c03aa953983ec7de2610e52007bbacf434a6b0a"},"cell_type":"markdown","source":"That's it. If you have downloaded word2vec then you can start using this code.  Here are few examples of what it does.\n\ncorrection('quikly') returns quickly\n\ncorrection('israil') returns israel\n\ncorrection('neighbour') returns neighbor"},{"metadata":{"_cell_guid":"d2c822e9-8aea-9b4d-1e87-538cb8dab4f2","_uuid":"7cf8f54043cfbc5a14b7f64b8e9fd42bcd65a2f5","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18a8fedb6393808d3ddef8002a3bbbc29cff0e85"},"cell_type":"code","source":"puncts = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', '#', '*', '+', '\\\\', '•',  '~', '@', '£', \n '·', '_', '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', '½', 'à', '…', \n '“', '★', '”', '–', '●', 'â', '►', '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', \n '▒', '：', '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', '‘', '∞', \n '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', '¹', '≤', '‡', '√', ]\n\ndef clean_text(x):\n    x = str(x)\n    for punct in puncts:\n        if punct in x:\n            x = x.replace(punct, f' {punct} ')\n    return x\n\ntrain[\"question_text\"] = train[\"question_text\"].apply(lambda x: x.lower())\ntrain[\"question_text\"] = train[\"question_text\"].apply(lambda x: clean_text(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db89f197220b57eb71295db5ef849b99e0bebbc7"},"cell_type":"code","source":"def build_vocab(texts):\n    sentences = texts.apply(lambda x: x.split()).values\n    vocab = {}\n    for sentence in sentences:\n        for word in sentence:\n            try:\n                vocab[word] += 1\n            except KeyError:\n                vocab[word] = 1\n    return vocab\n\nvocab = build_vocab(train.question_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15c877028b511a9f746a522fd4c7aa09201be2b6"},"cell_type":"code","source":"import heapq\nfrom operator import itemgetter\n\ntop_90k_words = dict(heapq.nlargest(90000, vocab.items(), key=itemgetter(1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b13239508c595f48945a4536afd322b7c4e409c6"},"cell_type":"code","source":"from multiprocessing import Pool\npool = Pool(4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5fa87ec959b83e7e07882b29dd39ff27fd849b8f"},"cell_type":"code","source":"corrected_words = pool.map(correction,list(top_90k_words.keys()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b2d58f18fe2fb0fdb67435525dd799e980f03aa"},"cell_type":"code","source":"for word,corrected_word in zip(top_90k_words,corrected_words):\n    if word!=corrected_word:\n        print(word,\":\",corrected_word)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c23af7703eca217019cb25411988ac2b2eb61a14"},"cell_type":"markdown","source":"It is not perfect but helps to understand the dataset much better. \nFor example:\n'baelish' is converted to Bullish\n'sansa' to salsa\n'tarly' to early\n'elon' to felon\n\nApparently doesn't understand the Game of thrones characters. Does that mean a lot of GOT characters are there?\n\nAlso you can try to run something like this after your usual text cleaning pipeline. Hope that helps to remove some spelling errors. \n"},{"metadata":{"_uuid":"39867cd0ed4dfdc25c3780cf96bff1a4201882a8"},"cell_type":"markdown","source":"If you like this notebook then please upvote (button at the top right)."}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}