{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cleaning and removing misspells from texts","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"As you know, to enhance the performance of a model you need to make sure you feed it with high-quality data.\n\n**What does 'high-quality' mean? \n\nWell, it means that you don't have misspelling errors, that your text is not polluted by artifacts or other errors.\n\nIn this short notebook, I show you two ways of cleaning your data.\n\n**Feel free to share your thoughts on my work ;)**\n\nSources:  \n- https://www.kaggle.com/shonenkov/hack-with-parallel-corpus\n- https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/147417 (Dave Lorenz's message)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Importing dependencies","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install -q pandarallel\n!pip install -q spacy \n!pip install -q spacy_cld\n!pip install -q pyspellchecker\n!python -m spacy download xx_ent_wiki_sm > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport os\nimport gc\n\nimport spacy\nfrom spacy_cld import LanguageDetector\nimport xx_ent_wiki_sm\n\nfrom spellchecker import SpellChecker\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport time\nimport random\nfrom tqdm.notebook import tqdm\ntqdm.pandas()\n\nimport re\nimport nltk\n\nfrom pandarallel import pandarallel\npandarallel.initialize(progress_bar=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Language score","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"nlp = xx_ent_wiki_sm.load()\nlanguage_detector = LanguageDetector()\nnlp.add_pipe(language_detector)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lang_score(text, lang):\n    try:\n        doc = nlp(str(text))\n        language_scores = doc._.language_scores\n        return language_scores.get(lang, 0)\n    except Exception:\n        return 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading data\n\ntrain1 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\ntrain1['lang'] = 'en'\n\ntrain_es = pd.read_csv('/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-es-cleaned.csv')\ntrain_es['lang'] = 'es'\n\ntrain_fr = pd.read_csv('/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-fr-cleaned.csv')\ntrain_fr['lang'] = 'fr'\n\ntrain_pt = pd.read_csv('/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-pt-cleaned.csv')\ntrain_pt['lang'] = 'pt'\n\ntrain_ru = pd.read_csv('/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-ru-cleaned.csv')\ntrain_ru['lang'] = 'ru'\n\ntrain_it = pd.read_csv('/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-it-cleaned.csv')\ntrain_it['lang'] = 'it'\n\ntrain_tr = pd.read_csv('/kaggle/input/jigsaw-train-multilingual-coments-google-api/jigsaw-toxic-comment-train-google-tr-cleaned.csv')\ntrain_tr['lang'] = 'tr'\n\ntrain2 = pd.read_csv(\"/kaggle/input/jigsaw-multilingual-toxic-comment-classification/jigsaw-unintended-bias-train.csv\")\ntrain2.toxic = train2.toxic.round().astype(int)\ntrain2['lang'] = 'en'\n\ntrain = pd.concat([\n    \n    train1[['comment_text', 'lang', 'toxic']],\n    train_es[['comment_text', 'lang', 'toxic']],\n    train_tr[['comment_text', 'lang', 'toxic']],\n    train_fr[['comment_text', 'lang', 'toxic']],\n    train_pt[['comment_text', 'lang', 'toxic']],\n    train_ru[['comment_text', 'lang', 'toxic']],\n    train_it[['comment_text', 'lang', 'toxic']],\n    train2[['comment_text', 'lang', 'toxic']]\n    \n]).sample(n=20000).reset_index(drop=True)\n\ndel train1, train_es, train_fr, train_pt, train_ru, train_it, train_tr, train2\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['lang_score'] = train.progress_apply(lambda x: get_lang_score(x['comment_text'], x['lang']), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(train['lang_score'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train[train['lang_score'] > 0.8]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Correct misspellings","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"spell = SpellChecker()\n\n# A quick example\nmisspelled = spell.unknown(['something', 'somegting', 'helo', 'fack', 'here', 'bijour'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Counting the number of spelling errors\n\ntrain['mispell_count'] = train['comment_text'].progress_apply(lambda x: len(spell.unknown(x.split())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['mispell_count'] < 100]['mispell_count'].hist(bins=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we have a lot of data, we can remove sentences with more than 20 mispells. But before that, let's have a look at the language distribution to make sure we don't remove too many sentences of the same language. Remember that SpellChecker is trained on some languages including Portuguese, Spanish, English, French.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train['lang'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train[train['mispell_count'] < 20]['lang'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As expected, Russian sentences might be a bit more discarded since Spellcheck might not be able to correctly evaluate Russian sentences.\n**Tips: Always remember to check your data distribution before doing any transformation ;)**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train[train['mispell_count'] < 20]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we can quickly replace the small artifacts \\n by nothing.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train['comment_text'] = train['comment_text'].apply(lambda x: x.replace('\\n', ' '))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We are done for this quick notebook. Hope it helped you clean your dataset. If so, don't hesitate to upvote the notebook ;)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}