{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n
NLP Preprocessing and Feature Extraction Methods A-Z
\n
\n
The Beginning to Intermediate Guide
\n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n# Introduction\n\nThis notebook's motivation is to create a ready-to-made all-in-one-place of NLP preprocessing and feature extraction techniques and codes. Furthermore, as we would not use all of those techniques simultaneously as it would depend on different specific NLP problems, each task was design as a separate module with its quick and straightforward explanation, then the implementation that we could pick out, plug-and-play independently and conveniently. We will mainly use the [Real or Not? NLP with Disaster Tweets](https://www.kaggle.com/c/nlp-getting-started) dataset for illustration and other dataset such as [Jigsaw Multilingual Toxic Comment Classification](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification), and [Toxic Comment Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) for some specific tasks too.\n\nThe NLP pipeline could be represent and below image and this notebook only focus on three stages: Text Cleaning, Pre-Processing and Feature Engineering/ Extraction.\n\n\n\nimage source: [Natural Language Processing Pipeline](https://towardsdatascience.com/natural-language-processing-pipeline-93df02ecd03f)\n\nThe paper [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067) has been inspired me a lot for this notebook and most of the definition of each tasks also could be found from the paper. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n* [Introduction](#Introduction)\n* [Read and explore data](#Read_and_explore_data)\n - [Importing Main Packages](#Importing_Main_Packages)\n - [Read the Data](#Read_the_Data)\n* [Text Cleaning](#Text_Cleaning)\n - [Capitalization/ Lower case](#Capitalization)\n - [Expand the Contractions](#Expand_the_Contractions)\n - [Noise Removal](#Noise_Removal)\n - [Remove URLs](#Remove_urls)\n - [Remove HTML tags](#Remove_HTML_tags)\n - [Remove Non-ASCII](#Remove_Non_ASCII)\n - [Remove special characters](#Remove_special_characters)\n - [Remove punctuations](#Remove_punctuations)\n - [Other Manual Text Cleaning Tasks](#Other_Manual_Text_Cleaning_Tasks)\n - [Replace the Typos, slang, acronyms or informal abbreviations](#Replace_Typos)\n - [Spelling correction](#Spelling_correction) \n* [Text Preprocessing](#Text_Preprocessing)\n - [Tokenization](#Tokenization)\n - [Remove Stop Words (or/and Frequent words/ Rare words)](#Remove_Stop_Words)\n - [Stemming](#Stemming)\n - [PorterStemmer](#PorterStemmer)\n - [SnowballStemmer](#SnowballStemmer)\n - [LancasterStemmer](#LancasterStemmer)\n - [Part of Speech Tagging (POS Tagging)](#POS_Tagging) \n - [Lemmatization](#Lemmatization)\n - [Lemmatization without POS Tagging](#Lemmatization_wo_pos)\n - [Lemmatization with POS tagging](#Lemmatization_w_pos)\n - [Other (Optional) Text Preprocessing Techniques:](#Other_Text_Preprocessing)\n - [Language Detection](#Language_Detection)\n* [Text Features Extraction](#Text_Features_Extraction)\n - [Weighted Words - Bag of Words (BoW)](#BoW)\n - [Frequency Vectors - CountVectorizer](#CountVectorizer)\n - [Term Frequency-Inverse Document Frequency](#TF_IDF)\n - [Word Embedding](#Word_Embedding)\n - [Basic Word Embedding Methods](#Basic_Word_Embedding)\n - [Word2Vec](#Word2Vec)\n - [Global Vectors for Word Representation](#GloVe)\n - [FastText](#FastText)\n - [Advanced Word Embedding Methods - Deep Contextualized Word Representations](#Advanced_methods)\n - [Bidirectional Encoder Representations from Transformers (BERT)](#BERT)\n - [Comparison of Feature Extraction Techniques](#Comparison)\n* [References](#References)\n - [Paper](#Paper)\n - [Books](#Books)\n - [Blogs/ Notebooks](#Blogs_Notebooks)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n# Read and explore data\n\n\n## Importing Main Packages\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time\nimport os\nimport sys\nimport warnings\nif not sys.warnoptions:\n warnings.simplefilter(\"ignore\")\n \nimport numpy as np\nimport pandas as pd\nimport sklearn\n\n# Libraries and packages for text (pre-)processing \nimport string\nimport re\nimport nltk\n\nprint(\"Python version:\", sys.version)\nprint(\"Version info.:\", sys.version_info)\nprint(\"pandas version:\", pd.__version__)\nprint(\"numpy version:\", np.__version__)\nprint(\"skearn version:\", sklearn.__version__)\nprint(\"re version:\", re.__version__)\nprint(\"nltk version:\", nltk.__version__)\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Read the Data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%time\n\n# read the csv file\ntrain_df = pd.read_csv(\"/kaggle/input/nlp-getting-started/train.csv\")\ndisplay(train_df.shape, train_df.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# some early explorations\n\ndisplay(train_df[~train_df[\"location\"].isnull()].head())\ndisplay(train_df[train_df[\"target\"] == 0][\"text\"].values[1])\ndisplay(train_df[train_df[\"target\"] == 1][\"text\"].values[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n# Text Cleaning:\n\n\n## Capitalization/ Lower case\nThe most common approach in text cleaning is capitalization or lower case due to the diversity of capitalization to form a sentence. This technique will project all words in text and document into the same feature space. However, it would also cause problems with exceptional cases such as the USA or UK, which could be solved by replacing typos, slang, acronyms or informal abbreviations technique.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train_df[\"text_clean\"] = train_df[\"text\"].apply(lambda x: x.lower())\ndisplay(train_df.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Expand the Contractions\nWe use the [contractions package](https://github.com/kootenpv/contractions) to expand the contraction in English such as we'll -> we will or we shouldn't've -> we should not have.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# Intall the contractions package - https://github.com/kootenpv/contractions\n!pip install contractions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time\nimport contractions\n\n# Test\ntest_text = \"\"\"\n Y'all can't expand contractions I'd think. I'd like to know how I'd done that! \n We're going to the zoo and I don't think I'll be home for dinner.\n Theyre going to the zoo and she'll be home for dinner.\n We should've do it in here but we shouldn't've eat it\n \"\"\"\nprint(\"Test: \", contractions.fix(test_text))\n\ntrain_df[\"text_clean\"] = train_df[\"text_clean\"].apply(lambda x: contractions.fix(x))\n\n# double check\nprint(train_df[\"text\"][67])\nprint(train_df[\"text_clean\"][67])\nprint(train_df[\"text\"][12])\nprint(train_df[\"text_clean\"][12])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Noise Removal \nText data could include various unnecessary characters or punctuation such as URLs, HTML tags, non-ASCII characters, or other special characters (symbols, emojis, and other graphic characters). \n\n\n### Remove URLs\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def remove_URL(text):\n \"\"\"\n Remove URLs from a sample string\n \"\"\"\n return re.sub(r\"https?://\\S+|www\\.\\S+\", \"\", text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# remove urls from the text\ntrain_df[\"text_clean\"] = train_df[\"text_clean\"].apply(lambda x: remove_URL(x))\n\n# double check\nprint(train_df[\"text\"][31])\nprint(train_df[\"text_clean\"][31])\nprint(train_df[\"text\"][37])\nprint(train_df[\"text_clean\"][37])\nprint(train_df[\"text\"][62])\nprint(train_df[\"text_clean\"][62])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Remove HTML tags\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def remove_html(text):\n \"\"\"\n Remove the html in sample text\n \"\"\"\n html = re.compile(r\"<.*?>|&([a-z0-9]+|#[0-9]{1,6}|#x[0-9a-f]{1,6});\")\n return re.sub(html, \"\", text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# remove html from the text\ntrain_df[\"text_clean\"] = train_df[\"text_clean\"].apply(lambda x: remove_html(x))\n\n# double check\nprint(train_df[\"text\"][62])\nprint(train_df[\"text_clean\"][62])\nprint(train_df[\"text\"][7385])\nprint(train_df[\"text_clean\"][7385])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Remove Non-ASCI:\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def remove_non_ascii(text):\n \"\"\"\n Remove non-ASCII characters \n \"\"\"\n return re.sub(r'[^\\x00-\\x7f]',r'', text) # or ''.join([x for x in text if x in string.printable]) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# remove non-ascii characters from the text\ntrain_df[\"text_clean\"] = train_df[\"text_clean\"].apply(lambda x: remove_non_ascii(x))\n\n# double check\nprint(train_df[\"text\"][38])\nprint(train_df[\"text_clean\"][38])\nprint(train_df[\"text\"][7586])\nprint(train_df[\"text_clean\"][7586])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Remove special characters: \nThe special characters could be symbols, emojis, and other graphic characters.\nWe use the \"Toxic Comment Classification Challenge\" dataset as the \"Real or Not? NLP with Disaster Tweets\" dataset do not have any special charaters in their text.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train_df_jtcc = pd.read_csv(\"/kaggle/input/jigsaw-toxic-comment-classification-challenge/train.csv.zip\")\nprint(train_df_jtcc.shape)\ntrain_df_jtcc.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def remove_special_characters(text):\n \"\"\"\n Remove special special characters, including symbols, emojis, and other graphic characters\n \"\"\"\n emoji_pattern = re.compile(\n '['\n u'\\U0001F600-\\U0001F64F' # emoticons\n u'\\U0001F300-\\U0001F5FF' # symbols & pictographs\n u'\\U0001F680-\\U0001F6FF' # transport & map symbols\n u'\\U0001F1E0-\\U0001F1FF' # flags (iOS)\n u'\\U00002702-\\U000027B0'\n u'\\U000024C2-\\U0001F251'\n ']+',\n flags=re.UNICODE)\n return emoji_pattern.sub(r'', text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time\n# remove non-ascii characters from the text\ntrain_df_jtcc[\"text_clean\"] = train_df_jtcc[\"comment_text\"].apply(lambda x: remove_special_characters(x))\ndisplay(train_df_jtcc.head())\n\n# double check\nprint(train_df_jtcc[\"comment_text\"][143])\nprint(train_df_jtcc[\"text_clean\"][143])\nprint(train_df_jtcc[\"comment_text\"][189])\nprint(train_df_jtcc[\"text_clean\"][189])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Saving disk space\ndel train_df_jtcc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Remove punctuations:\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def remove_punct(text):\n \"\"\"\n Remove the punctuation\n \"\"\"\n# return re.sub(r'[]!\"$%&\\'()*+,./:;=#@?[\\\\^_`{|}~-]+', \"\", text)\n return text.translate(str.maketrans('', '', string.punctuation))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# remove punctuations from the text\ntrain_df[\"text_clean\"] = train_df[\"text_clean\"].apply(lambda x: remove_punct(x))\n\n# double check\nprint(train_df[\"text\"][5])\nprint(train_df[\"text_clean\"][5])\nprint(train_df[\"text\"][7597])\nprint(train_df[\"text_clean\"][7597])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Other Manual Text Cleaning Tasks: \n\nOther techniques could be considered and manually processed case by case: \n - Replace the Unicode character with equivalent ASCII character (instead of removing)\n - Replace the entity references with their actual symbols instead of removing as HTML tags\n - Replace the Typos, slang, acronyms or informal abbreviations - depend on different situations or main topics of the NLP such as finance or medical topics.\n - List out all the hashtags/ usernames then replace with equivalent words\n - Replace the emoticon/ emoji with equivalant word meaning such as \":)\" with \"smile\" \n - Spelling correction\n\n\n### Replace the Typos, slang, acronyms or informal abbreviations: \n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def other_clean(text):\n \"\"\"\n Other manual text cleaning techniques\n \"\"\"\n # Typos, slang and other\n sample_typos_slang = {\n \"w/e\": \"whatever\",\n \"usagov\": \"usa government\",\n \"recentlu\": \"recently\",\n \"ph0tos\": \"photos\",\n \"amirite\": \"am i right\",\n \"exp0sed\": \"exposed\",\n \"<3\": \"love\",\n \"luv\": \"love\",\n \"amageddon\": \"armageddon\",\n \"trfc\": \"traffic\",\n \"16yr\": \"16 year\"\n }\n\n # Acronyms\n sample_acronyms = { \n \"mh370\": \"malaysia airlines flight 370\",\n \"okwx\": \"oklahoma city weather\",\n \"arwx\": \"arkansas weather\", \n \"gawx\": \"georgia weather\", \n \"scwx\": \"south carolina weather\", \n \"cawx\": \"california weather\",\n \"tnwx\": \"tennessee weather\",\n \"azwx\": \"arizona weather\", \n \"alwx\": \"alabama weather\",\n \"usnwsgov\": \"united states national weather service\",\n \"2mw\": \"tomorrow\"\n }\n\n \n # Some common abbreviations \n sample_abbr = {\n \"$\" : \" dollar \",\n \"€\" : \" euro \",\n \"4ao\" : \"for adults only\",\n \"a.m\" : \"before midday\",\n \"a3\" : \"anytime anywhere anyplace\",\n \"aamof\" : \"as a matter of fact\",\n \"acct\" : \"account\",\n \"adih\" : \"another day in hell\",\n \"afaic\" : \"as far as i am concerned\",\n \"afaict\" : \"as far as i can tell\",\n \"afaik\" : \"as far as i know\",\n \"afair\" : \"as far as i remember\",\n \"afk\" : \"away from keyboard\",\n \"app\" : \"application\",\n \"approx\" : \"approximately\",\n \"apps\" : \"applications\",\n \"asap\" : \"as soon as possible\",\n \"asl\" : \"age, sex, location\",\n \"atk\" : \"at the keyboard\",\n \"ave.\" : \"avenue\",\n \"aymm\" : \"are you my mother\",\n \"ayor\" : \"at your own risk\", \n \"b&b\" : \"bed and breakfast\",\n \"b+b\" : \"bed and breakfast\",\n \"b.c\" : \"before christ\",\n \"b2b\" : \"business to business\",\n \"b2c\" : \"business to customer\",\n \"b4\" : \"before\",\n \"b4n\" : \"bye for now\",\n \"b@u\" : \"back at you\",\n \"bae\" : \"before anyone else\",\n \"bak\" : \"back at keyboard\",\n \"bbbg\" : \"bye bye be good\",\n \"bbc\" : \"british broadcasting corporation\",\n \"bbias\" : \"be back in a second\",\n \"bbl\" : \"be back later\",\n \"bbs\" : \"be back soon\",\n \"be4\" : \"before\",\n \"bfn\" : \"bye for now\",\n \"blvd\" : \"boulevard\",\n \"bout\" : \"about\",\n \"brb\" : \"be right back\",\n \"bros\" : \"brothers\",\n \"brt\" : \"be right there\",\n \"bsaaw\" : \"big smile and a wink\",\n \"btw\" : \"by the way\",\n \"bwl\" : \"bursting with laughter\",\n \"c/o\" : \"care of\",\n \"cet\" : \"central european time\",\n \"cf\" : \"compare\",\n \"cia\" : \"central intelligence agency\",\n \"csl\" : \"can not stop laughing\",\n \"cu\" : \"see you\",\n \"cul8r\" : \"see you later\",\n \"cv\" : \"curriculum vitae\",\n \"cwot\" : \"complete waste of time\",\n \"cya\" : \"see you\",\n \"cyt\" : \"see you tomorrow\",\n \"dae\" : \"does anyone else\",\n \"dbmib\" : \"do not bother me i am busy\",\n \"diy\" : \"do it yourself\",\n \"dm\" : \"direct message\",\n \"dwh\" : \"during work hours\",\n \"e123\" : \"easy as one two three\",\n \"eet\" : \"eastern european time\",\n \"eg\" : \"example\",\n \"embm\" : \"early morning business meeting\",\n \"encl\" : \"enclosed\",\n \"encl.\" : \"enclosed\",\n \"etc\" : \"and so on\",\n \"faq\" : \"frequently asked questions\",\n \"fawc\" : \"for anyone who cares\",\n \"fb\" : \"facebook\",\n \"fc\" : \"fingers crossed\",\n \"fig\" : \"figure\",\n \"fimh\" : \"forever in my heart\", \n \"ft.\" : \"feet\",\n \"ft\" : \"featuring\",\n \"ftl\" : \"for the loss\",\n \"ftw\" : \"for the win\",\n \"fwiw\" : \"for what it is worth\",\n \"fyi\" : \"for your information\",\n \"g9\" : \"genius\",\n \"gahoy\" : \"get a hold of yourself\",\n \"gal\" : \"get a life\",\n \"gcse\" : \"general certificate of secondary education\",\n \"gfn\" : \"gone for now\",\n \"gg\" : \"good game\",\n \"gl\" : \"good luck\",\n \"glhf\" : \"good luck have fun\",\n \"gmt\" : \"greenwich mean time\",\n \"gmta\" : \"great minds think alike\",\n \"gn\" : \"good night\",\n \"g.o.a.t\" : \"greatest of all time\",\n \"goat\" : \"greatest of all time\",\n \"goi\" : \"get over it\",\n \"gps\" : \"global positioning system\",\n \"gr8\" : \"great\",\n \"gratz\" : \"congratulations\",\n \"gyal\" : \"girl\",\n \"h&c\" : \"hot and cold\",\n \"hp\" : \"horsepower\",\n \"hr\" : \"hour\",\n \"hrh\" : \"his royal highness\",\n \"ht\" : \"height\",\n \"ibrb\" : \"i will be right back\",\n \"ic\" : \"i see\",\n \"icq\" : \"i seek you\",\n \"icymi\" : \"in case you missed it\",\n \"idc\" : \"i do not care\",\n \"idgadf\" : \"i do not give a damn fuck\",\n \"idgaf\" : \"i do not give a fuck\",\n \"idk\" : \"i do not know\",\n \"ie\" : \"that is\",\n \"i.e\" : \"that is\",\n \"ifyp\" : \"i feel your pain\",\n \"IG\" : \"instagram\",\n \"iirc\" : \"if i remember correctly\",\n \"ilu\" : \"i love you\",\n \"ily\" : \"i love you\",\n \"imho\" : \"in my humble opinion\",\n \"imo\" : \"in my opinion\",\n \"imu\" : \"i miss you\",\n \"iow\" : \"in other words\",\n \"irl\" : \"in real life\",\n \"j4f\" : \"just for fun\",\n \"jic\" : \"just in case\",\n \"jk\" : \"just kidding\",\n \"jsyk\" : \"just so you know\",\n \"l8r\" : \"later\",\n \"lb\" : \"pound\",\n \"lbs\" : \"pounds\",\n \"ldr\" : \"long distance relationship\",\n \"lmao\" : \"laugh my ass off\",\n \"lmfao\" : \"laugh my fucking ass off\",\n \"lol\" : \"laughing out loud\",\n \"ltd\" : \"limited\",\n \"ltns\" : \"long time no see\",\n \"m8\" : \"mate\",\n \"mf\" : \"motherfucker\",\n \"mfs\" : \"motherfuckers\",\n \"mfw\" : \"my face when\",\n \"mofo\" : \"motherfucker\",\n \"mph\" : \"miles per hour\",\n \"mr\" : \"mister\",\n \"mrw\" : \"my reaction when\",\n \"ms\" : \"miss\",\n \"mte\" : \"my thoughts exactly\",\n \"nagi\" : \"not a good idea\",\n \"nbc\" : \"national broadcasting company\",\n \"nbd\" : \"not big deal\",\n \"nfs\" : \"not for sale\",\n \"ngl\" : \"not going to lie\",\n \"nhs\" : \"national health service\",\n \"nrn\" : \"no reply necessary\",\n \"nsfl\" : \"not safe for life\",\n \"nsfw\" : \"not safe for work\",\n \"nth\" : \"nice to have\",\n \"nvr\" : \"never\",\n \"nyc\" : \"new york city\",\n \"oc\" : \"original content\",\n \"og\" : \"original\",\n \"ohp\" : \"overhead projector\",\n \"oic\" : \"oh i see\",\n \"omdb\" : \"over my dead body\",\n \"omg\" : \"oh my god\",\n \"omw\" : \"on my way\",\n \"p.a\" : \"per annum\",\n \"p.m\" : \"after midday\",\n \"pm\" : \"prime minister\",\n \"poc\" : \"people of color\",\n \"pov\" : \"point of view\",\n \"pp\" : \"pages\",\n \"ppl\" : \"people\",\n \"prw\" : \"parents are watching\",\n \"ps\" : \"postscript\",\n \"pt\" : \"point\",\n \"ptb\" : \"please text back\",\n \"pto\" : \"please turn over\",\n \"qpsa\" : \"what happens\", #\"que pasa\",\n \"ratchet\" : \"rude\",\n \"rbtl\" : \"read between the lines\",\n \"rlrt\" : \"real life retweet\", \n \"rofl\" : \"rolling on the floor laughing\",\n \"roflol\" : \"rolling on the floor laughing out loud\",\n \"rotflmao\" : \"rolling on the floor laughing my ass off\",\n \"rt\" : \"retweet\",\n \"ruok\" : \"are you ok\",\n \"sfw\" : \"safe for work\",\n \"sk8\" : \"skate\",\n \"smh\" : \"shake my head\",\n \"sq\" : \"square\",\n \"srsly\" : \"seriously\", \n \"ssdd\" : \"same stuff different day\",\n \"tbh\" : \"to be honest\",\n \"tbs\" : \"tablespooful\",\n \"tbsp\" : \"tablespooful\",\n \"tfw\" : \"that feeling when\",\n \"thks\" : \"thank you\",\n \"tho\" : \"though\",\n \"thx\" : \"thank you\",\n \"tia\" : \"thanks in advance\",\n \"til\" : \"today i learned\",\n \"tl;dr\" : \"too long i did not read\",\n \"tldr\" : \"too long i did not read\",\n \"tmb\" : \"tweet me back\",\n \"tntl\" : \"trying not to laugh\",\n \"ttyl\" : \"talk to you later\",\n \"u\" : \"you\",\n \"u2\" : \"you too\",\n \"u4e\" : \"yours for ever\",\n \"utc\" : \"coordinated universal time\",\n \"w/\" : \"with\",\n \"w/o\" : \"without\",\n \"w8\" : \"wait\",\n \"wassup\" : \"what is up\",\n \"wb\" : \"welcome back\",\n \"wtf\" : \"what the fuck\",\n \"wtg\" : \"way to go\",\n \"wtpa\" : \"where the party at\",\n \"wuf\" : \"where are you from\",\n \"wuzup\" : \"what is up\",\n \"wywh\" : \"wish you were here\",\n \"yd\" : \"yard\",\n \"ygtr\" : \"you got that right\",\n \"ynk\" : \"you never know\",\n \"zzz\" : \"sleeping bored and tired\"\n }\n \n sample_typos_slang_pattern = re.compile(r'(?\n\n### Spelling Correction\nSpelling correction could also be considered an optional preprocessing task as the social media text data is often are typos or mistyped. However, the spelling correction output should be carefully double-checked with the original text input as it could be a mistake.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from textblob import TextBlob\nprint(\"Test: \", TextBlob(\"sleapy and tehre is no plaxe I'm gioong to.\").correct())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n# Text Preprocessing:\n\n\n## Tokenization\nTokenization is a common technique that split a sentence into tokens, where a token could be characters, words, phrases, symbols, or other meaningful elements. By breaking sentences into smaller chunks, that would help to investigate the words in a sentence and also the subsequent steps in the NLP pipeline, such as stemming. \n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# Tokenizing the tweet base texts.\nfrom nltk.tokenize import word_tokenize\n\ntrain_df['tokenized'] = train_df['text_clean'].apply(word_tokenize)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Remove Stop Words (or/and Frequent words/ Rare words):\nStop words are common words in any language that occur with a high frequency but do not deliver meaningful information for the whole sentence. For example, {“a”, “about”, “above”, “across”, “after”, “afterward”, “again”, ...} can be considered as stop words. Traditionally, we could remove all of them in the text preprocessing stage. However, refer to the example from the [Natural Language Processing in Action](https://www.manning.com/books/natural-language-processing-in-action) book: \n> * Mark reported to the CEO\n> * Suzanne reported as the CEO to the board \n\n> In your NLP pipeline, you might create 4-grams such as reported to the CEO and reported as the CEO. If you remove the stop words from the 4-grams, both examples would be reduced to \"reported CEO\", and you would lack the information about the professional hierarchy. In the first example, Mark could have been an assistant to the CEO, whereas in the second example Suzanne was the CEO reporting to the board. Unfortunately, retaining the stop words within your pipeline creates another problem: it increases the length of the n-grams required to make use of these connections formed by the otherwise meaningless stop words. This issue forces us to retain at least 4-grams if you want to avoid the ambiguity of the human resources example.\n> Designing a filter for stop words depends on your particular application.\n\nIn short, removing stop words is a common method in NLP text preprocessing, whereas, it needs to be experimented carefully depending on different situations. \n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Removing stopwords.\nnltk.download(\"stopwords\")\nfrom nltk.corpus import stopwords\n\nstop = set(stopwords.words('english'))\ntrain_df['stopwords_removed'] = train_df['tokenized'].apply(lambda x: [word for word in x if word not in stop])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Stemming\nStemming is a process of extracting a root word - identifying a common stem among various forms (e.g., singular and plural noun form) of a word, for example, the words \"gardening\", \"gardener\" or \"gardens\" share the same stem, garden. Stemming uproots suffixes from words to merge words with similar meanings under their standard stem.\n\nThere are three major stemming algorithms in use nowadays:\n- **Porter** - PorterStemmer()): This stemming algorithm is an older one. It’s from the 1980s and its main concern is removing the common endings to words so that they can be resolved to a common form. It’s not too complex and development on it is frozen. Typically, it’s a nice starting basic stemmer, but it’s not really advised to use it for any production/complex application. Instead, it has its place in research as a nice, basic stemming algorithm that can guarantee reproducibility. It also is a very gentle stemming algorithm when compared to others.\n\n- **Snowball** - LancasterStemmer(): This algorithm is also known as the Porter2 stemming algorithm. It is almost universally accepted as better than the Porter stemmer, even being acknowledged as such by the individual who created the Porter stemmer. That being said, it is also more aggressive than the Porter stemmer. A lot of the things added to the Snowball stemmer were because of issues noticed with the Porter stemmer. There is about a 5% difference in the way that Snowball stems versus Porter.\n\n- **Lancaster** - SnowballStemmer(): Just for fun, the Lancaster stemming algorithm is another algorithm that you can use. This one is the most aggressive stemming algorithm of the bunch. However, if you use the stemmer in NLTK, you can add your own custom rules to this algorithm very easily. It’s a good choice for that. One complaint around this stemming algorithm though is that it sometimes is overly aggressive and can really transform words into strange stems. Just make sure it does what you want it to before you go with this option!\n\nsource: http://hunterheidenreich.com/blog/stemming-lemmatization-what/\n\n\n### PorterStemmer\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from nltk.stem import PorterStemmer\n\ndef porter_stemmer(text):\n \"\"\"\n Stem words in list of tokenized words with PorterStemmer\n \"\"\"\n stemmer = nltk.PorterStemmer()\n stems = [stemmer.stem(i) for i in text]\n return stems","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\ntrain_df['porter_stemmer'] = train_df['stopwords_removed'].apply(lambda x: porter_stemmer(x))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### SnowballStemmer\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from nltk.stem import SnowballStemmer\n\ndef snowball_stemmer(text):\n \"\"\"\n Stem words in list of tokenized words with SnowballStemmer\n \"\"\"\n stemmer = nltk.SnowballStemmer(\"english\")\n stems = [stemmer.stem(i) for i in text]\n return stems","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\ntrain_df['snowball_stemmer'] = train_df['stopwords_removed'].apply(lambda x: snowball_stemmer(x))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### LancasterStemmer\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from nltk.stem import LancasterStemmer\n\ndef lancaster_stemmer(text):\n \"\"\"\n Stem words in list of tokenized words with LancasterStemmer\n \"\"\"\n stemmer = nltk.LancasterStemmer()\n stems = [stemmer.stem(i) for i in text]\n return stems","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\ntrain_df['lancaster_stemmer'] = train_df['stopwords_removed'].apply(lambda x: lancaster_stemmer(x))\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Part of Speech Tagging (POS Tagging):\nPart of speech tagging (POS tagging) distinguishes the part of speech (noun, verb, adjective, and etc.) of each word in the text. This is the critical stage for many NLP applications since, by identifying the POS of a word, we can infer its contextual meaning. The NLTK packages offer different POS Tagging algorithms, and in this notebook, we use the combination version of them.\n\n- pos_tag/ DefaultTagger\n- UnigramTagger\n- BigramTagger\n- Could also be a combination of the bigram tagger, unigram tagger, and default tagger (source: https://www.nltk.org/book/ch05.html) \n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from nltk.corpus import wordnet\nfrom nltk.corpus import brown\n\nwordnet_map = {\"N\":wordnet.NOUN, \n \"V\":wordnet.VERB, \n \"J\":wordnet.ADJ, \n \"R\":wordnet.ADV\n }\n \ntrain_sents = brown.tagged_sents(categories='news')\nt0 = nltk.DefaultTagger('NN')\nt1 = nltk.UnigramTagger(train_sents, backoff=t0)\nt2 = nltk.BigramTagger(train_sents, backoff=t1)\n\ndef pos_tag_wordnet(text, pos_tag_type=\"pos_tag\"):\n \"\"\"\n Create pos_tag with wordnet format\n \"\"\"\n pos_tagged_text = t2.tag(text)\n \n # map the pos tagging output with wordnet output \n pos_tagged_text = [(word, wordnet_map.get(pos_tag[0])) if pos_tag[0] in wordnet_map.keys() else (word, wordnet.NOUN) for (word, pos_tag) in pos_tagged_text ]\n return pos_tagged_text","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"pos_tag_wordnet(train_df['stopwords_removed'][2])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\ntrain_df['combined_postag_wnet'] = train_df['stopwords_removed'].apply(lambda x: pos_tag_wordnet(x))\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Lemmatization:\nAccording to the [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf) book:\n> Lemmatization is the task of determining that two words have the same root, despite their surface differences. The words am, are, and is have the shared lemma be; the words dinner and dinners both have the lemma dinner. Lemmatizing each of these forms to the same lemma will let us find all mentions of words in Russian like Moscow. The lemmatized form of a sentence like He is reading detective stories would thus be He be read detective story.\n\nand the book [Natural Language Processing in Action](https://www.manning.com/books/natural-language-processing-in-action):\n> Some lemmatizers use the word’s part of speech (POS) tag in addition to its spelling to help improve accuracy. The POS tag for a word indicates its role in the grammar of a phrase or sentence. For example, the noun POS is for words that refer to “people, places, or things” within a phrase. An adjective POS is for a word that modifies or describes a noun. A verb refers to an action. The POS of a word in isolation cannot be determined. The context of a word must be known for its POS to be identified. So some advanced lemmatizers can’t be run-on words in isolation.\n\nFor example, the \"good\", \"better\" or \"best\" is lemmatized into good and the verb \"gardening\" should be lemmatized to \"to garden\", while the \"garden\" and \"gardener\" are both different lemmas. In this notebook, we will also explore on both lemmatize on without POS-Tagging and POS-Tagging examples.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from nltk.stem import WordNetLemmatizer\n\ndef lemmatize_word(text):\n \"\"\"\n Lemmatize the tokenized words\n \"\"\"\n\n lemmatizer = WordNetLemmatizer()\n lemma = [lemmatizer.lemmatize(word, tag) for word, tag in text]\n return lemma","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Lemmatization without POS Tagging:\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\n# Test without POS Tagging\nlemmatizer = WordNetLemmatizer()\n\ntrain_df['lemmatize_word_wo_pos'] = train_df['stopwords_removed'].apply(lambda x: [lemmatizer.lemmatize(word) for word in x])\ntrain_df['lemmatize_word_wo_pos'] = train_df['lemmatize_word_wo_pos'].apply(lambda x: [word for word in x if word not in stop])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"print(train_df[\"combined_postag_wnet\"][8])\nprint(train_df[\"lemmatize_word_wo_pos\"][8])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Lemmatization with POS Tagging:\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\n# Test with POS Tagging\nlemmatizer = WordNetLemmatizer()\n\ntrain_df['lemmatize_word_w_pos'] = train_df['combined_postag_wnet'].apply(lambda x: lemmatize_word(x))\ntrain_df['lemmatize_word_w_pos'] = train_df['lemmatize_word_w_pos'].apply(lambda x: [word for word in x if word not in stop]) # double check to remove stop words\ntrain_df['lemmatize_text'] = [' '.join(map(str, l)) for l in train_df['lemmatize_word_w_pos']] # join back to text\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Comparing the output of Lemmatization on non-POS-Tagging and POS-Tagging output. We can see in the original text, the word \\\" happening\\\" is a verb and was corrected assigned as a verb by POS-tagging stage, then Lemmatize accurately with back as \\\"happen\\\" but lemmatized without-POS-tagging resulted in \\\"happening\\\" is not correct. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df[\"text\"][8])\nprint(train_df[\"combined_postag_wnet\"][8])\nprint(train_df[\"lemmatize_word_wo_pos\"][8])\nprint(train_df[\"lemmatize_word_w_pos\"][8])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Comparison between original text and the lammatized text:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train_df[\"text\"][0], train_df[\"lemmatize_text\"][0])\ndisplay(train_df[\"text\"][5], train_df[\"lemmatize_text\"][5])\ndisplay(train_df[\"text\"][10], train_df[\"lemmatize_text\"][10])\ndisplay(train_df[\"text\"][15], train_df[\"lemmatize_text\"][15])\ndisplay(train_df[\"text\"][20], train_df[\"lemmatize_text\"][20])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Other (Optional) Text Preprocessing Techniques:\n- language detection\n- Code mixing and transliteration\n\n\n### Language Detection:\nWe will use the package [polyglot](https://github.com/aboSamoor/polyglot) for language detection\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"# Install the main polygot and other neccesary packages\n!pip install pyicu\n!pip install pycld2\n!pip install polyglot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will use the \"Jigsaw Multilingual Toxic Comment Classification\" dataset for this case as the dataset is multilingual","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"train_df_jmtc = pd.read_csv(\"../input/jigsaw-multilingual-toxic-comment-classification/jigsaw-toxic-comment-train.csv\")\nprint(train_df_jmtc.shape)\ntrain_df_jmtc.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \nfrom polyglot.detect import Detector\n\ndef get_language(text):\n return Detector(\"\".join(x for x in text if x.isprintable()), quiet=True).languages[0].name\n\ntrain_df_jmtc[\"lang\"] = train_df_jmtc[\"comment_text\"].apply(lambda x: get_language(x))\n\n#Test\ndisplay(train_df_jmtc[train_df_jmtc[\"lang\"] == \"de\"].head())\nprint(train_df_jmtc[\"comment_text\"][823])\nprint(train_df_jmtc[\"comment_text\"][8130])\nprint(train_df_jmtc[\"comment_text\"][14511])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save disk space\ndel train_df_jmtc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### code mixing and transliteration:\nThis situation should be considered in case of multilingual text such as the mixed up between English and other languages.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n# Text Features Extraction:\n\n\n## Weighted Words - Bag of Words (BoW) - Bag of n-grams:\n* N-gram is a sequence that contains n-elements (characters, words, etc). A single word such a \"apple\", \"orange\" is a Uni-gram; hence, \"red apple\" \"big orange\" is bi-gram and \"red ripped apple\", \"big orange bag\" is tri-gram. \n* Bags of words: Vectors of word counts or frequencies \n* Bags of n-grams: Counts of word pairs (bigrams), triplets (trigrams), and so on\n\n> The bag-of-words/ bag-of-n-gram model is a reduced and simplified representation of a text document from selected parts of the text, based on specific criteria, such as word frequency.\n> \n> In a BoW, a body of text, such as a document or a sentence, is thought of like a bag of words. Lists of words are created in the BoW process. These words in a matrix are not sentences which structure sentences and grammar, and the semantic relationship between these words are ignored in their collection and construction. The words are often representative of the content of a sentence. While grammar and order of appearance are ignored, multiplicity is counted and may be used later to determine the focus points of the documents.\n> \n> Example:\n> Document\n> \n> “As the home to UVA’s recognized undergraduate and graduate degree programs in systems engineering. In the UVA Department of Systems and Information Engineering, our students are exposed to a wide range of range”\n> \n> Bag-of-Words (BoW):\n> {“As”, “the”, “home”, “to”, “UVA’s”, “recognized”, “undergraduate”, “and”, “graduate”, “degree”, “program”, “in”, “systems”, “engineering”, “in”, “Department”, “Information”,“students”, “ ”,“are”, “exposed”, “wide”, “range” }\n> \n> Bag-of-Feature (BoF)\n> Feature = {1,1,1,3,2,1,2,1,2,3,1,1,1,2,1,1,1,1,1,1}\n\n(source:[Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067))\n\n\n### Frequency Vectors - CountVectorizer:\nWe will implement the Bag of Words/ Bag of n-grams text representation via sklearn - CountVectorizer function.\nThe code will test with a sample corpus of the first five sentence of the dataset, then print out the output of uni-gram, bi-gram and tri-gram. Finaly, we also run on the whole dataset.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\n\ndef cv(data, ngram = 1, MAX_NB_WORDS = 75000):\n count_vectorizer = CountVectorizer(ngram_range = (ngram, ngram), max_features = MAX_NB_WORDS)\n emb = count_vectorizer.fit_transform(data).toarray()\n print(\"count vectorize with\", str(np.array(emb).shape[1]), \"features\")\n return emb, count_vectorizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"def print_out(emb, feat, ngram, compared_sentence=0):\n print(ngram,\"bag-of-words: \")\n print(feat.get_feature_names(), \"\\n\")\n print(ngram,\"bag-of-feature: \")\n print(test_cv_1gram.vocabulary_, \"\\n\")\n print(\"BoW matrix:\")\n print(pd.DataFrame(emb.transpose(), index = feat.get_feature_names()).head(), \"\\n\")\n print(ngram,\"vector example:\")\n print(train_df[\"lemmatize_text\"][compared_sentence])\n print(emb[compared_sentence], \"\\n\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"test_corpus = train_df[\"lemmatize_text\"][:5].tolist()\nprint(\"The test corpus: \", test_corpus, \"\\n\")\n\ntest_cv_em_1gram, test_cv_1gram = cv(test_corpus, ngram=1)\nprint_out(test_cv_em_1gram, test_cv_1gram, ngram=\"Uni-gram\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_cv_em_2gram, test_cv_2gram = cv(test_corpus, ngram=2)\nprint_out(test_cv_em_2gram, test_cv_2gram, ngram=\"Bi-gram\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_cv_em_3gram, test_cv_3gram = cv(test_corpus, ngram=3)\nprint_out(test_cv_em_2gram, test_cv_2gram, ngram=\"Tri-gram\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time \n\n# implement into the whole dataset\ntrain_df_corpus = train_df[\"lemmatize_text\"].tolist()\ntrain_df_em_1gram, vc_1gram = cv(train_df_corpus, 1)\ntrain_df_em_2gram, vc_2gram = cv(train_df_corpus, 2)\ntrain_df_em_3gram, vc_3gram = cv(train_df_corpus, 3)\n\nprint(len(train_df_corpus))\nprint(train_df_em_1gram.shape)\nprint(train_df_em_2gram.shape)\nprint(train_df_em_3gram.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df_em_1gram, train_df_em_2gram, train_df_em_3gram","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Term Frequency-Inverse Document Frequency (TF-IDF):\n> The Inverse Document Frequency (IDF) as a method to be used in conjunction with term frequency in order to lessen the effect of implicitly common words in the corpus. IDF assigns a higher weight to words with either high or low frequencies term in the document. This combination of TF and IDF is well known as Term Frequency-Inverse document frequency (TF-IDF). The mathematical representation of the weight of a term in a document by TF-IDF is given in Equation: \n> $$ W(d,t) = TF(d,t) * log \\frac{N}{df(t)}$$\n> Here N is the number of documents and $df(t)$ is the number of documents containing the term t in the corpus. The first term in the equation improves the recall while the second term improves the precision of the word embedding. Although TF-IDF tries to overcome the problem of common terms in the document, it still suffers from some other descriptive limitations. Namely, TF-IDF cannot account for the similarity between the words in the document since each word is independently presented as an index. However, with the development of more complex models in recent years, new methods, such as word embedding, have been presented that can incorporate concepts such as similarity of words and part of speech tagging.\n\n(source: [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067))\n\nWe also implement the TF-IDF via sklearn TfidfVectorizer function, the experiments are similar to the previous [Frequency Vectors - CountVectorizer](#CountVectorizer) section\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\n\ndef TFIDF(data, ngram = 1, MAX_NB_WORDS = 75000):\n tfidf_x = TfidfVectorizer(ngram_range = (ngram, ngram), max_features = MAX_NB_WORDS)\n emb = tfidf_x.fit_transform(data).toarray()\n print(\"tf-idf with\", str(np.array(emb).shape[1]), \"features\")\n return emb, tfidf_x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"test_corpus = train_df[\"lemmatize_text\"][:5].tolist()\nprint(\"The test corpus: \", test_corpus, \"\\n\")\n\ntest_tfidf_em_1gram, test_tfidf_1gram = TFIDF(test_corpus, ngram=1)\nprint_out(test_tfidf_em_1gram, test_tfidf_1gram, ngram=\"Uni-gram\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_tfidf_em_2gram, test_tfidf_2gram = TFIDF(test_corpus, ngram=2)\nprint_out(test_tfidf_em_2gram, test_tfidf_2gram, ngram=\"Bi-gram\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_tfidf_em_3gram, test_tfidf_3gram = TFIDF(test_corpus, ngram=3)\nprint_out(test_tfidf_em_3gram, test_tfidf_3gram, ngram=\"Tri-gram\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time \n\n# implement into the whole dataset\ntrain_df_corpus = train_df[\"lemmatize_text\"].tolist()\ntrain_df_tfidf_1gram, tfidf_1gram = TFIDF(train_df_corpus, 1)\ntrain_df_tfidf_2gram, tfidf_2gram = TFIDF(train_df_corpus, 2)\ntrain_df_tfidf_3gram, tfidf_3gram = TFIDF(train_df_corpus, 3)\n\nprint(len(train_df_corpus))\nprint(train_df_tfidf_1gram.shape)\nprint(train_df_tfidf_1gram.shape)\nprint(train_df_tfidf_1gram.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df_tfidf_1gram, train_df_tfidf_2gram, train_df_tfidf_3gram","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n## Word Embedding:\n\n> **Word vectors** are numerical vector representations of word semantics, or meaning, including literal and implied meaning. So word vectors can capture the connotation of words, like “peopleness,” “animalness,” “placeness,” “thingness,” and even “conceptness.” And they combine all that into a dense vector (no zeros) of floating point values. This dense vector enables queries and logical reasoning.\n\n(source: [Natural Language Processing in Action](https://www.manning.com/books/natural-language-processing-in-action))\n\n> Even though we have syntactic word representations, it does not mean that the model captures the semantics meaning of the words. On the other hand, bag-of-word models do not respect the semantics of the word. For example, words “airplane”, “aeroplane”, “plane”, and “aircraft” are often used in the same context. However, the vectors corresponding to these words are orthogonal in the bag-of-words model. This issue presents a serious problem to understanding sentences within the model. The other problem in the bag-of-word is that the order of words in the phrase is not respected. The n-gram does not solve this problem so a similarity needs to be found for each word in the sentence. Many researchers worked on word embedding to solve this problem. The Word2Vec propose a simple single-layer architecture based on the inner product between two word vectors.\n\n> Word embedding is a feature learning technique in which each word or phrase from the vocabulary is mapped to a N dimension vector of real numbers. Various word embedding methods have been proposed to translate unigrams into understandable input for machine learning algorithms. This work focuses on Word2Vec, GloVe, and FastText, three of the most common methods that have been successfully used for deep learning techniques.\n\n(source: [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067))\n\n\n### Basic Word Embedding Methods:\n\n\n#### Word2Vec:\n\n[T. Mikolov et al.](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) presented the Word2vec in 2013, which learns the meaning of words merely by processing a large corpus of unlabeled text. The Word2Vec approach uses shallow neural networks with two hidden layers, continuous bag-of-words (CBOW), and the Skip-gram model to create a high dimension vector for each word. This unsupervised nature of Word2vec is what makes it so powerful. The world is full of unlabeled, uncategorized, unstructured natural language text.\n\nWe will implement the Word2vec via gensim libary with the pre-trained word vectors on the dataset Google News corpus (source: https://code.google.com/archive/p/word2vec/) and see the embedding output on the sample sentence from the our dataset. \n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time \n\nimport gensim\nprint(\"gensim version:\", gensim.__version__)\n\nword2vec_path = \"../input/googlenewsvectorsnegative300/GoogleNews-vectors-negative300.bin\"\n\n# we only load 200k most common words from Google News corpus \nword2vec_model = gensim.models.KeyedVectors.load_word2vec_format(word2vec_path, binary=True, limit=200000) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compare the similarity between \"cat\" vs. \"kitten\" and \"cat\" vs. \"cats\"","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(word2vec_model.similarity('cat', 'kitten'))\nprint(word2vec_model.similarity('cat', 'cats'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"def get_average_vec(tokens_list, vector, generate_missing=False, k=300):\n \"\"\"\n Calculate average embedding value of sentence from each word vector\n \"\"\"\n \n if len(tokens_list)<1:\n return np.zeros(k)\n \n if generate_missing:\n vectorized = [vector[word] if word in vector else np.random.rand(k) for word in tokens_list]\n else:\n vectorized = [vector[word] if word in vector else np.zeros(k) for word in tokens_list]\n \n length = len(vectorized)\n summed = np.sum(vectorized, axis=0)\n averaged = np.divide(summed, length)\n return averaged\n\ndef get_embeddings(vectors, text, generate_missing=False, k=300):\n \"\"\"\n create the sentence embedding\n \"\"\"\n embeddings = text.apply(lambda x: get_average_vec(x, vectors, generate_missing=generate_missing, k=k))\n return list(embeddings)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\nembeddings_word2vec = get_embeddings(word2vec_model, train_df[\"lemmatize_text\"], k=300)\n\nprint(\"Embedding matrix size\", len(embeddings_word2vec), len(embeddings_word2vec[0]))\nprint(\"The sentence: \\\"%s\\\" got embedding values: \" % train_df[\"lemmatize_text\"][0])\nprint(embeddings_word2vec[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del embeddings_word2vec","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n#### Global Vectors for Word Representation (GloVe):\n> Another powerful word embedding technique that has been used for text classification is [Global Vectors (GloVe)](https://nlp.stanford.edu/pubs/glove.pdf). The approach is very similar to the Word2Vec method, where each word is presented by a high dimension vector and trained based on the surrounding words over a huge corpus. The pre-trained word embedding used in many works is based on 400,000 vocabularies trained over Wikipedia 2014 and Gigaword 5 as the corpus and 50 dimensions for word presentation. GloVe also provides other pre-trained word vectorizations with 100, 200, 300 dimensions which are trained over even bigger corpora, including Twitter content.\n\n(source: [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067))\n\nWe will create our GloVe's sentence embeddings via gensim libary with the pre-trained word vectors on the dataset from Wikipedia 2014 + Gigaword 5 (source: https://github.com/stanfordnlp/GloVe) and see the embedding output on the sample sentence from the our dataset. \n\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time \n\nfrom gensim.scripts.glove2word2vec import glove2word2vec\n\nglove_input_file = \"../input/glove6b/glove.6B.300d.txt\"\nword2vec_output_file = \"glove.6B.100d.txt.word2vec\"\nglove2word2vec(glove_input_file, word2vec_output_file)\n\n# we only load 200k most common words from Google New corpus \nglove_model = gensim.models.KeyedVectors.load_word2vec_format(word2vec_output_file, binary=False, limit=200000) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compare the similarity between \"cat\" vs. \"kitten\" and \"cat\" vs. \"cats\" from GloVe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(glove_model.similarity('cat', 'kitten'))\nprint(glove_model.similarity('cat', 'cats'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\nembeddings_glove = get_embeddings(glove_model, train_df[\"lemmatize_text\"], k=300)\n\nprint(\"Embedding matrix size\", len(embeddings_glove), len(embeddings_glove[0]))\nprint(\"The sentence: \\\"%s\\\" got embedding values: \" % train_df[\"lemmatize_text\"][0])\nprint(embeddings_glove[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del embeddings_glove","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n#### FastText:\n> Many other word embedding representations ignore the morphology of words by assigning a distinct vector to each word ([Enriching Word Vectors with Subword Information](https://arxiv.org/abs/1607.04606)). Facebook AI Research lab released a novel technique to solve this issue by introducing a new word embedding method called FastText. Each word, w, is represented as a bag of character n-gram. For example, given the word “introduce” and n = 3, FastText will produce the following representation composed of character tri-grams: < in, int, ntr, tro, rod, odu, duc, uce, ce >\n> Note that the sequence , corresponding to the word here is different from the tri-gram “int” from the word introduce.\n\n(source: [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067))\n\nWe will create our FastText's sentence embeddings via gensim libary with the pre-trained word vectors from the Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset (source: https://fasttext.cc/docs/en/english-vectors.html) and see the embedding output on the sample sentence from the our dataset. \n\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time \n\nfrom gensim.models.fasttext import FastText\n\nfasttext_path = \"../input/fasttext-wikinews/wiki-news-300d-1M.vec\"\nfasttext_model = gensim.models.KeyedVectors.load_word2vec_format(fasttext_path, binary=False, limit=200000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compare the similarity between \"cat\" vs. \"kitten\" and \"cat\" vs. \"cats\" from FastText","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(fasttext_model.similarity('cat', 'kitten'))\nprint(fasttext_model.similarity('cat', 'cats'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"embeddings_fasttext = get_embeddings(fasttext_model, train_df[\"lemmatize_text\"], k=300)\n\nprint(\"Embedding matrix size\", len(embeddings_fasttext), len(embeddings_fasttext[0]))\nprint(\"The sentence: \\\"%s\\\" got embedding values: \" % train_df[\"lemmatize_text\"][0])\nprint(embeddings_fasttext[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del embeddings_fasttext","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Advanced Word Embedding Methods - Deep Contextualized Word Representations: \n\n\n#### Bidirectional Encoder Representations from Transformers (BERT):\n> BERT is a deep learning model that has given state-of-the-art results on a wide variety of natural language processing tasks. It stands for Bidirectional Encoder Representations for Transformers. It has been pre-trained on Wikipedia and BooksCorpus and requires task-specific fine-tuning.\n\n> Lets understand BERT by breaking BERT abbreviation:\n> * **Bidirectional**: BERT takes whole text passage as input and reads passage in both direction to understand the meaning of each word.\n> * **Transformers**: BERT is based on a Deep Transformer network. Transformer network is a type of network that can process efficiently long texts by using attention. An attention is a mechanism to learn contextual relations between words (or sub-words) in a text.\n> * **Encoder Representation**: Originally Transformer includes two separate mechanisms — an encoder that reads the text input and a decoder that produces a prediction for the task, since BERT’s goal is to generate a language model only the encoder mechanism is necessary hence 'encoder representation'\n\n> BERT is a multi-layer bidirectional Transformer encoder. There are two models introduced in the paper.\n> * BERT base – 12 layers (transformer blocks), 12 attention heads, and 110 million parameters.\n> * BERT Large – 24 layers, 16 attention heads and, 340 million parameters.\n\n\n> How BERT performs Bidirectional training?\n> \n> BERT uses following two prediction models simultaneously with the goal of minimizing the combined loss function of the two strategies:\n> \n> * **Masked Language Model**: Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence.\n> * **Next Sentence Prediction**: The model receives pairs of sentences as input and learns to predict if the second sentence in the pair is the subsequent sentence in the original document. During training, 50% of the inputs are a pair in which the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence. The assumption is that the random sentence will be disconnected from the first sentence.\n\nResources and further reading on BERT's explanation could be found in the great Kaggle notebooks and Blogs here:\n* https://www.kaggle.com/abhinand05/bert-for-humans-tutorial-baseline-version-2\n* https://www.kaggle.com/ratan123/in-depth-guide-to-google-s-bert\n* https://www.kaggle.com/kksienc/comprehensive-nlp-tutorial-3-bert\n* https://yashuseth.blog/2019/06/12/bert-explained-faqs-understand-bert-working/\n\nWe will create our sentence embeddings by BERT's pre-trained word vectors (Uncased) via Tensorflow (source: https://github.com/google-research/bert) and see the embedding output on the sample sentence from the our dataset. Noted that we will use the BERT isself tonkenizer. \n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"%time \n\nimport tensorflow_hub as hub\n\n# download the tonkenizer \n!wget --quiet https://raw.githubusercontent.com/tensorflow/models/master/official/nlp/bert/tokenization.py\nimport tokenization","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"module_url = \"https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/1\"\nbert_layer = hub.KerasLayer(module_url, trainable=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def bert_encode(texts, tokenizer, max_len=512):\n all_tokens = []\n all_masks = []\n all_segments = []\n \n for text in texts:\n text = tokenizer.tokenize(text)\n \n text = text[:max_len-2]\n input_sequence = [\"[CLS]\"] + text + [\"[SEP]\"]\n pad_len = max_len - len(input_sequence)\n \n tokens = tokenizer.convert_tokens_to_ids(input_sequence)\n tokens += [0] * pad_len\n pad_masks = [1] * len(input_sequence) + [0] * pad_len\n segment_ids = [0] * max_len\n \n all_tokens.append(tokens)\n all_masks.append(pad_masks)\n all_segments.append(segment_ids)\n \n return np.array(all_tokens), np.array(all_masks), np.array(all_segments)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%time \n\nvocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()\ndo_lower_case = bert_layer.resolved_object.do_lower_case.numpy()\ntokenizer = tokenization.FullTokenizer(vocab_file, do_lower_case)\n\nbert_input = bert_encode(train_df[\"text\"].values, tokenizer, max_len=300)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Embedding tensor size\", len(bert_input), len(bert_input[0]), len(bert_input[0][0]))\nprint(\"The sentence: \\\"%s\\\" got embedding values: \" % train_df[\"lemmatize_text\"][0])\nprint(bert_input[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## Comparison of Feature Extraction Techniques\nPlease refer to the below table as the Comparison between Feature Extraction Techniques, thanks to the paper [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067) for all of their awesome works.\n\n[Back To Table of Contents](#top_section)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"| Model \t| Advantages \t| Limitation \t|\n|-------------------------------------\t|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\t|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\t|\n| Weighted Words \t| * Easy to compute
* Easy to compute the similarity between 2 documents using it
* Basic metric to extract the most descriptive terms in a document
* Works with an unknown word (e.g., New words in languages) \t| * It does not capture the position in the text (syntactic)
* It does not capture meaning in the text (semantics)
* Common words effect on the results (e.g., “am”, “is”, etc.) \t|\n| TF-IDF \t| * Easy to compute
* Easy to compute the similarity between 2 documents using it
* Basic metric to extract the most descriptive terms in a document
* Common words do not affect the results due to IDF (e.g., “am”, “is”, etc.) \t| * It does not capture the position in the text (syntactic)
* It does not capture meaning in the text (semantics) \t|\n| Word2Vec \t| * It captures the position of the words in the text (syntactic)
* It captures meaning in the words (semantics) \t| * It cannot capture the meaning of the word from the text (fails to capture polysemy)
* It cannot capture out-of-vocabulary words from corpus \t|\n| GloVe (Pre-Trained) \t| * It captures the position of the words in the text (syntactic)
* It captures meaning in the words (semantics)
* Trained on huge corpus \t| * It cannot capture the meaning of the word from the text (fails to capture polysemy)
* Memory consumption for storage
* It cannot capture out-of-vocabulary words from corpus \t|\n| GloVe (Trained) \t| * It is very straightforward, e.g., to enforce the word vectors to capture sub-linear relationships in the vector space (performs better than Word2vec)
* Lower weight for highly frequent word pairs, such as stop words like “am”, “is”, etc. Will not dominate training progress \t| * Memory consumption for storage
* Needs huge corpus to learn
* It cannot capture out-of-vocabulary words from the corpus
* It cannot capture the meaning of the word from the text (fails to capture polysemy) \t|\n| FastText \t| * Works for rare words (rare in their character n-grams which are still shared with other words
* Solves out of vocabulary words with n-gram in character level \t| * It cannot capture the meaning of the word from the text (fails to capture polysemy)
* Memory consumption for storage
* Computationally is more expensive in comparing with GloVe and Word2Vec \t|\n| Contextualized Word Representations \t| * It captures the meaning of the word from the text (incorporates context, handling polysemy) \t| * Memory consumption for storage
* Improves performance notably on downstream tasks. Computationally is more expensive in comparison to others
* Needs another word embedding for all LSTM and feedforward layers
* It cannot capture out-of-vocabulary words from a corpus
* Works only sentence and document level (it cannot work for individual word level) \t|","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n\n# References:\n\n## Paper:\n* [Text Classification Algorithms: A Survey](https://arxiv.org/abs/1904.08067)\n\n\n## Books:\n* [Natural Language Processing in Action](https://www.manning.com/books/natural-language-processing-in-action)\n* [Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf)\n\n\n## Blogs/ Notebooks:\n* https://www.kaggle.com/abhinand05/bert-for-humans-tutorial-baseline-version-2\n* https://www.kaggle.com/amar09/text-pre-processing-and-feature-extraction \n* https://www.kaggle.com/ashishpatel26/beginner-to-intermediate-nlp-tutorial \n* https://www.kaggle.com/ashutosh3060/nlp-basic-feature-creation-and-preprocessing \n* https://www.kaggle.com/datafan07/disaster-tweets-nlp-eda-bert-with-transformers \n* https://www.kaggle.com/gunesevitan/nlp-with-disaster-tweets-eda-cleaning-and-bert \n* https://www.kaggle.com/kksienc/comprehensive-nlp-tutorial-3-bert\n* https://www.kaggle.com/liananapalkova/simply-about-word2vec \n* https://www.kaggle.com/ratan123/in-depth-guide-to-google-s-bert \n* https://www.kaggle.com/sudalairajkumar/getting-started-with-text-preprocessing\n* https://www.kaggle.com/vbmokin/nlp-eda-bag-of-words-tf-idf-glove-bert \n* https://www.kdnuggets.com/2018/08/practitioners-guide-processing-understanding-text-2.html \n* https://yashuseth.blog/2019/06/12/bert-explained-faqs-understand-bert-working/\n* https://gist.github.com/MrEliptik/b3f16179aa2f530781ef8ca9a16499af \n* https://github.com/hundredblocks/concrete_NLP_tutorial/blob/master/NLP_notebook.ipynb \n* https://machinelearningmastery.com/gentle-introduction-bag-words-model/ \n* https://towardsdatascience.com/natural-language-processing-pipeline-93df02ecd03f\n\n\n### I really appreciate your feedbacks, there would be some areas can be fixed and improved.\n## If you liked my work please Upvote!\n[Back To Table of Contents](#top_section)","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}