{"cells":[{"metadata":{"_uuid":"521178d5e7c0e427d5aff12df5d6b64aacc495ec"},"cell_type":"markdown","source":"# Completely lost? Annotated Questions Preprocessing \n\nHi all, this kernel is aimed to help you guys really understand the nuts and bolts. I've shamelessly copied the code and some of the text from Siddharth Yadav's kernel - [Analyzing Quora for the Insinceres](https://www.kaggle.com/thebrownviking20/analyzing-quora-for-the-insinceres) and tried to explain every bit of the code - what it does and why its there. \n\nLet me know If you like it, by leaving a comment and upvoting. \n\nHave fun.\n\n\n\n\n# About the dataset\nAn existential problem for any major website today is how to handle toxic and divisive content. Quora wants to tackle this problem head-on to keep their platform a place where users can feel safe sharing their knowledge with the world.\n\nQuora is a platform that empowers people to learn from each other. On Quora, people can ask questions and connect with others who contribute unique insights and quality answers. A key challenge is to weed out insincere questions -- those founded upon false premises, or that intend to make a statement rather than look for helpful answers.\n\nIn this competition, Kagglers will develop models that identify and flag insincere questions. To date, Quora has employed both machine learning and manual review to address this problem. With your help, they can develop more scalable methods to detect toxic and misleading content.\n\nHere's your chance to combat online trolls at scale. Help Quora uphold their policy of “Be Nice, Be Respectful” and continue to be a place for sharing and growing the world’s knowledge."},{"metadata":{"_uuid":"7f9987565f5f217b163e688bddca83c31c669bdf"},"cell_type":"markdown","source":"###  Import Python libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Usual imports\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\nimport string\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.decomposition import NMF, LatentDirichletAllocation, TruncatedSVD\nfrom statistics import *\nfrom sklearn.feature_extraction.text import CountVectorizer\nimport concurrent.futures\nimport time\nimport pyLDAvis.sklearn\nfrom pylab import bone, pcolor, colorbar, plot, show, rcParams, savefig\nimport textstat\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Plotly based imports for visualization\nfrom plotly import tools\nimport plotly.plotly as py\nfrom plotly.offline import init_notebook_mode, iplot\ninit_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.figure_factory as ff\n\n# spaCy based imports\nimport spacy\nfrom spacy.lang.en.stop_words import STOP_WORDS\nfrom spacy.lang.en import English","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0324cd5e736d25e78313d02da0461aa8a1a0e5f2"},"cell_type":"markdown","source":"###  Print content of inputs archive from kaggle"},{"metadata":{"trusted":true,"_uuid":"50f026e93d903c8a520b8bdd99df1ffaf9d1d8cd"},"cell_type":"code","source":"%matplotlib inline\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b87b84e1b44f9def3ffde70000fe1efcc4759288"},"cell_type":"markdown","source":"### Read Training Data set"},{"metadata":{"trusted":true,"_uuid":"0e2da97e5cd6e24ce093f3e6ccfa5bb514b5fb49"},"cell_type":"code","source":"quora_train = pd.read_csv(\"../input/train.csv\")\nquora_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de7bb80363a8760e2f4544f2c9eddae37d60a408"},"cell_type":"markdown","source":"## Pre-processing\nIf you're working with a lot of text, you'll eventually want to know more about it. For example, what's it about? What do the words mean in context? Who is doing what to whom? Which texts are similar to each other?\nThe process of converting data to something a computer can understand is referred to as pre-processing. \n\nIn this case, we will tokenize the data, get rid on useless words and standardize lower and upper case based on the LEMMA of each word (Lemma is the base form of the word - for example, the lemma of \"is\" is \"be\" - to understand this better, read more  [here](https://spacy.io/usage/linguistic-features)\n\n###   Tokenization - Segmenting text into words, punctuations marks etc.\n\nAltough it is possible to work with the questions as a whole, tokenization allows us to compare different features such as similar words, etc. It also allows us to disconsider punctuation and stop words.\nThe code bellow creates a function that will then be used to parse the question text using the spacy library.\nTo understand how this works, take a look [here](https://spacy.io/usage/spacy-101)"},{"metadata":{"_uuid":"48f3f77e666866d0eceb13b9de7b6f2e926d4c90"},"cell_type":"markdown","source":"Here is an example of how the tokenization process works:"},{"metadata":{"trusted":true,"_uuid":"c93093852b31ab5080781ea5fa8c462b98f6986c"},"cell_type":"code","source":"sentence=\"I love it, when David writes Great looking code\"\nparser = English() # Defines the parse sapcy will use\nmytokens = parser(sentence)\nprint(mytokens)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c40d4fb39d13ccdb1cfc2e52de7e3fe86ef801d8"},"cell_type":"markdown","source":"The following code transforms each word in mytokens into the lemma of the word. There is an issue with creating the leema of pronouns like I and his - the function returns \"-PRON-\" for those words (read more about this [here](https://github.com/explosion/spaCy/issues/962). As a workarround, we ask for the lemma of the word. If the lemma is \"-PRON-\", then we ask for the word in itself in lowercase.\n<blockquote class=\"imgur-embed-pub\" lang=\"en\" data-id=\"a/q1EEPRt\"><a href=\"//imgur.com/q1EEPRt\"></a></blockquote><script async src=\"//s.imgur.com/min/embed.js\" charset=\"utf-8\"></script>                                        "},{"metadata":{"trusted":true,"_uuid":"444c45a86f5f5fb802e44ae67b1c124a337b9fdd"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c0c1c46932bc1c928ea6d2e1337bd8c55e5a941"},"cell_type":"code","source":"mytokens = [ word.lemma_.lower().strip() if word.lemma_ != \"-PRON-\" else word.lower_ for word in mytokens ]\nprint(mytokens)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3a8881e93954f9adc9e0db5b9f432f0d9e32aae"},"cell_type":"markdown","source":"Now, all thats left is to filter out stop words and punctuation. The idea is to use only words with semantical meaning in the analysis. "},{"metadata":{"trusted":true,"_uuid":"266004bd2b90b8e9701ff9b35b120bafddbd6efe"},"cell_type":"code","source":"punctuations = string.punctuation  #gets a list of puctuations carachters from the string library\nstopwords = list(STOP_WORDS) #gets a list of stop words - words that usually have little meaning in the phrase\nmytokens = [ word for word in mytokens if word not in stopwords and word not in punctuations ]\nprint(mytokens)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"02d040d9e824373871a236f53dcb4c8c109a3fe8"},"cell_type":"markdown","source":"After all that, we join the tokens to recreate the question. For that, we will use the join function. "},{"metadata":{"trusted":true,"_uuid":"25e32eeb628b9472b46f12064f501c9fdc8c2292"},"cell_type":"code","source":"mytokens = \" \".join([i for i in mytokens]) #go over each token in \"mytokens\" and add it to a string. Use a space (\" \".) to separate the words in the new string.\nprint(mytokens)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ff64ce7eae500bb61c726575b9e0c1d11fa1cca"},"cell_type":"markdown","source":"Thats it. We have a processed string that can be fed into a model. But.... We did it only once for a specific phrase. Now its time to encapsulate all that into a function that can be used again to process the whole questions file."},{"metadata":{"trusted":true,"_uuid":"239f186671cd2dcea82cf3e34b994eb29fab5279"},"cell_type":"code","source":"# SpaCy Parser for questions\npunctuations = string.punctuation\nstopwords = list(STOP_WORDS)\nparser = English()\n\ndef spacy_tokenizer(sentence): #Create a function called spacy_tokenizer that takes \"sentence\" as an argument and returns the processed sentence \n    mytokens = parser(sentence)\n    mytokens = [ word.lemma_.lower().strip() if word.lemma_ != \"-PRON-\" else word.lower_ for word in mytokens ]\n    mytokens = [ word for word in mytokens if word not in stopwords and word not in punctuations ]\n    mytokens = \" \".join([i for i in mytokens])\n    return mytokens","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a5c4716d9fe075adcdd7d98870b1e1263b436c4"},"cell_type":"markdown","source":"The code bellow Uses the function we just defined to proccess the tex of all quora questions.\nThe first line uses the tqdm function to ask the computer to show the progress of whats happening. It helps to know that the computer is actually doing something and not just stuck... Read more about this function [here](https://pypi.org/project/tqdm/).\nThe second line creates a file called sincere_questions that includes the processed result of all questions from the file quora_train where target=0 (sincere questions).\nThe third line creates a file called insincere_questions that includes the parsed result of all questions from the file quora_train where target=1 (insincere questions).\nYou can understand the code as\n<blockquote class=\"imgur-embed-pub\" lang=\"en\" data-id=\"F2TEG96\"><a href=\"//imgur.com/F2TEG96\">Understanding Python code -  Spacy tokenizer</a></blockquote><script async src=\"//s.imgur.com/min/embed.js\" charset=\"utf-8\"></script>:\n"},{"metadata":{"trusted":true,"_uuid":"57656d248a9e4c3972c812c61d68c8d215704732"},"cell_type":"code","source":"tqdm.pandas()\nsincere_questions = quora_train[\"question_text\"][quora_train[\"target\"] == 0].progress_apply(spacy_tokenizer)\ninsincere_questions = quora_train[\"question_text\"][quora_train[\"target\"] == 1].progress_apply(spacy_tokenizer)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2140035bf3df84d6489036653a6eaccb18911b7b"},"cell_type":"markdown","source":"## Hasta La Vista, Baby"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}