{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\nfrom wordcloud import WordCloud as wc\nfrom nltk.corpus import stopwords\nimport matplotlib.pylab as pylab\nimport matplotlib.pyplot as plt\nfrom pandas import get_dummies\nimport matplotlib as mpl\nimport seaborn as sns\nimport pandas as pd\nimport numpy as np\nimport matplotlib\nimport warnings\nimport sklearn\nimport string\nimport scipy\nimport numpy\nimport nltk\nimport json\nimport sys\nimport csv\nimport os\n\nprint('matplotlib: {}'.format(matplotlib.__version__))\nprint('sklearn: {}'.format(sklearn.__version__))\nprint('scipy: {}'.format(scipy.__version__))\nprint('seaborn: {}'.format(sns.__version__))\nprint('pandas: {}'.format(pd.__version__))\nprint('numpy: {}'.format(np.__version__))\nprint('Python: {}'.format(sys.version))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8645feedee1145c2df1268a697b0b8773858ad1a"},"cell_type":"code","source":"sns.set(style='white', context='notebook', palette='deep')\npylab.rcParams['figure.figsize'] = 12,8\nwarnings.filterwarnings('ignore')\nmpl.style.use('ggplot')\nsns.set_style('white')\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c5048b61a4837c8826551c8871609973ebbe3847"},"cell_type":"markdown","source":"
\n## 5-5 NLTK\nIn this kernel, we use the NLTK library So, before we begin the next step, we will first introduce this library.\nThe Natural Language Toolkit (NLTK) is one of the leading platforms for working with human language data and Python, the module NLTK is used for natural language processing. NLTK is literally an acronym for Natural Language Toolkit. With it you can tokenize words and sentences.\nNLTK is a library of Python that can mine (scrap and upload data) and analyse very large amounts of textual data using computational methods.\n"},{"metadata":{"_kg_hide-input":true,"_uuid":"adadeb7a83d0bc711a779948197c40841b10f1ca","trusted":true},"cell_type":"code","source":"from nltk.tokenize import sent_tokenize, word_tokenize\n \ndata = \"All work and no play makes jack a dull boy, all work and no play\"\nprint(word_tokenize(data))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e49407d9fe96b86c9851fbd7065ebfb218281687"},"cell_type":"markdown","source":"
\nAll of them are words except the comma. Special characters are treated as separate tokens.\n\n## 5-5-1 Tokenizing sentences\nThe same principle can be applied to sentences. Simply change the to sent_tokenize()\nWe have added two sentences to the variable data:"},{"metadata":{"_kg_hide-input":true,"_uuid":"ec9e6c715a1d49b2813c934fb4405ccec77884a1","trusted":true},"cell_type":"code","source":"from nltk.tokenize import sent_tokenize, word_tokenize\n \ndata = \"All work and no play makes jack dull boy. All work and no play makes jack a dull boy.\"\nprint(sent_tokenize(data))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75bb691fdb4982097ee8eb59ae930c1d81074afa"},"cell_type":"markdown","source":"
\n## 5-5-2 NLTK and arrays\nIf you wish to you can store the words and sentences in arrays"},{"metadata":{"_kg_hide-input":true,"_uuid":"fde02d4189b0a52b7f919ac0fa0643d84ebacaf7","trusted":true},"cell_type":"code","source":"from nltk.tokenize import sent_tokenize, word_tokenize\n \ndata = \"All work and no play makes jack dull boy. All work and no play makes jack a dull boy.\"\n \nphrases = sent_tokenize(data)\nwords = word_tokenize(data)\n \nprint(phrases)\nprint(words)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7fad127cb8ed99cc063b98b3391645263737958"},"cell_type":"markdown","source":"
\n## 5-5-3 NLTK stop words\nStop words are basically a set of commonly used words in any language, not just English. The reason why stop words are critical to many applications is that, if we remove the words that are very commonly used in a given language, we can focus on the important words instead.[12]"},{"metadata":{"_kg_hide-input":true,"_uuid":"3357ec158943478d584c392bb7702fe7e6d4b355","trusted":true},"cell_type":"code","source":"from nltk.tokenize import sent_tokenize, word_tokenize\nfrom nltk.corpus import stopwords\n \ndata = \"All work and no play makes jack dull boy. All work and no play makes jack a dull boy.\"\nstopWords = set(stopwords.words('english'))\nwords = word_tokenize(data)\nwordsFiltered = []\n \nfor w in words:\n if w not in stopWords:\n wordsFiltered.append(w)\n \nprint(wordsFiltered)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"581a7ba2ce1ae5dae6c36d54f8999af838c7b80c"},"cell_type":"markdown","source":"A module has been imported:\n\n"},{"metadata":{"_uuid":"2cb63648a5f138fe779744f0c52d570f30f84b13","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from nltk.corpus import stopwords","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7528080723ea540729e78b6e135475a870a5618"},"cell_type":"markdown","source":"We get a set of English stop words using the line:\n\n"},{"metadata":{"_uuid":"6fbe468728072fb1883e064d0c1e892259fb1c0c","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"stopWords = set(stopwords.words('english'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43843cdaccbe961422631c13d982e13bf25607c6"},"cell_type":"markdown","source":"The returned list stopWords contains 153 stop words on my computer.\nYou can view the length or contents of this array with the lines:"},{"metadata":{"_uuid":"53582f8f5ae2871e2cbba4542fc38965b61a5012","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(len(stopWords))\nprint(stopWords)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d25fed9ed1fd0016cea56de8e71b010a0d3176c3"},"cell_type":"markdown","source":"We create a new list called wordsFiltered which contains all words which are not stop words.\nTo create it we iterate over the list of words and only add it if its not in the stopWords list."},{"metadata":{"_uuid":"3b28823a9862bb263183c6d62a4e91286bfe8d30","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for w in words:\n if w not in stopWords:\n wordsFiltered.append(w)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8546a903a5b3916bca4feb96116dd00db1fc51c0"},"cell_type":"markdown","source":"
\n## 5-5-4 NLTK – stemming\nStart by defining some words:"},{"metadata":{"_uuid":"f3a8e1427f235183fc5b9656a15a1cdb9befb55b","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"words = [\"game\",\"gaming\",\"gamed\",\"games\"]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4a6ea3737dbb032d08c896f24cc6555b7b516274","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from nltk.stem import PorterStemmer\nfrom nltk.tokenize import sent_tokenize, word_tokenize","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4bd4f0bc00227a77cfe734575ebfa7d124d333d4"},"cell_type":"markdown","source":"And stem the words in the list using:"},{"metadata":{"_uuid":"12b5f99ca390ed21e68862e5eb5968d31e3858ef","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from nltk.stem import PorterStemmer\nfrom nltk.tokenize import sent_tokenize, word_tokenize\n\nwords = [\"game\",\"gaming\",\"gamed\",\"games\"]\nps = PorterStemmer()\n \nfor word in words:\n print(ps.stem(word))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0af11da2319b2be2643ad0003ebbc207067dc34"},"cell_type":"markdown","source":"
\n## 5-5-5 NLTK speech tagging\nThe module NLTK can automatically tag speech.\nGiven a sentence or paragraph, it can label words such as verbs, nouns and so on.\n\nNLTK – speech tagging example\nThe example below automatically tags words with a corresponding class."},{"metadata":{"_uuid":"fe586131bec724c1901a441b56753c1d47562483","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import nltk\nfrom nltk.tokenize import PunktSentenceTokenizer\n \ndocument = 'Whether you\\'re new to programming or an experienced developer, it\\'s easy to learn and use Python.'\nsentences = nltk.sent_tokenize(document) \nfor sent in sentences:\n print(nltk.pos_tag(nltk.word_tokenize(sent)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22e469c3b880de5c4f00a609192e74d97d22436a"},"cell_type":"markdown","source":"We can filter this data based on the type of word:"},{"metadata":{"_uuid":"8edca0d46e25f4d8dba85c454bb70299b7c1e112","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import nltk\nfrom nltk.corpus import state_union\nfrom nltk.tokenize import PunktSentenceTokenizer\n \ndocument = 'Today the Netherlands celebrates King\\'s Day. To honor this tradition, the Dutch embassy in San Francisco invited me to'\nsentences = nltk.sent_tokenize(document) \n \ndata = []\nfor sent in sentences:\n data = data + nltk.pos_tag(nltk.word_tokenize(sent))\n \nfor word in data: \n if 'NNP' in word[1]: \n print(word)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"2a498652935827a033d0bcb7df1d28c068f44a25","trusted":true},"cell_type":"code","source":"sns.set(style='white', context='notebook', palette='deep')\npylab.rcParams['figure.figsize'] = 12,8\nwarnings.filterwarnings('ignore')\nmpl.style.use('ggplot')\nsns.set_style('white')\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aafdf15f9c86416ee36ca83d198bb2f212ba9208"},"cell_type":"markdown","source":"
\n## 5-5-6 Natural Language Processing – prediction\nWe can use natural language processing to make predictions. Example: Given a product review, a computer can predict if its positive or negative based on the text. In this article you will learn how to make a prediction program based on natural language processing."},{"metadata":{"_uuid":"38cc515716212f2236fc7aa8fafbe93b40e2561a"},"cell_type":"markdown","source":"
\n### 5-5-5-6-1 nlp prediction example\nGiven a name, the classifier will predict if it’s a male or female.\n\nTo create our analysis program, we have several steps:\n\n1. Data preparation\n1. Feature extraction\n1. Training\n1. Prediction\n1. Data preparation\nThe first step is to prepare data. We use the names set included with nltk."},{"metadata":{"trusted":true,"_uuid":"a10a4b67460c9e0779234416316e5a163389a017"},"cell_type":"code","source":"from nltk.corpus import names\n \n# Load data and training \nnames = ([(name, 'male') for name in names.words('male.txt')] + \n\t [(name, 'female') for name in names.words('female.txt')])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4b2604b6496cf2cd03d7fef102fbdbf8e6dcd4b"},"cell_type":"markdown","source":"This dataset is simply a collection of tuples. To give you an idea of what the dataset looks like:"},{"metadata":{"trusted":true,"_uuid":"77ab1eabc4089ac1154bba9a822374480dba6ab7","_kg_hide-input":true},"cell_type":"code","source":"[(u'Aaron', 'male'), (u'Abbey', 'male'), (u'Abbie', 'male')]\n[(u'Zorana', 'female'), (u'Zorina', 'female'), (u'Zorine', 'female')]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c9cf2b995e1bc74a682e78fd301266f115c5577d"},"cell_type":"markdown","source":"You can define your own set of tuples if you wish, its simply a list containing many tuples.\n\nFeature extraction\nBased on the dataset, we prepare our feature. The feature we will use is the last letter of a name:\nWe define a featureset using:"},{"metadata":{"_uuid":"107b843e7e5d63f2d21b76919b26d6ff9514a6f0"},"cell_type":"markdown","source":"featuresets = [(gender_features(n), g) for (n,g) in names]\nand the features (last letters) are extracted using:"},{"metadata":{"trusted":true,"_uuid":"c3b127d6025c488eae76ce93ffce43536bdfea78"},"cell_type":"code","source":"def gender_features(word): \n return {'last_letter': word[-1]}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e889ae78839261ae4d419996a2c7c315ac9b4186"},"cell_type":"markdown","source":"Training and prediction\nWe train and predict using:"},{"metadata":{"trusted":true,"_uuid":"ed10f9ce265b40d77ab52a9bfe0e391d95ad1c75","_kg_hide-input":true},"cell_type":"code","source":"import nltk.classify.util\nfrom nltk.classify import NaiveBayesClassifier\nfrom nltk.corpus import names\n \ndef gender_features(word): \n return {'last_letter': word[-1]} \n \n# Load data and training \nnames = ([(name, 'male') for name in names.words('male.txt')] + \n\t [(name, 'female') for name in names.words('female.txt')])\n \nfeaturesets = [(gender_features(n), g) for (n,g) in names] \ntrain_set = featuresets\nclassifier = nltk.NaiveBayesClassifier.train(train_set) \n \n# Predict\nprint(names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(featuresets)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(classifier.classify(gender_features('Frank')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(classifier.classify(gender_features('Frankie')))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"375a50f4fe205a7b62c9ccb1aaec7cab4ad7ed09"},"cell_type":"markdown","source":"If you want to give the name during runtime, change the last line to:\n"},{"metadata":{"trusted":true,"_uuid":"3ff919bd79c13fda55e929f2bf4e3dd874253414","_kg_hide-input":true},"cell_type":"code","source":"# Predict, you can change name\nname = 'Sarah'\nprint(classifier.classify(gender_features(name)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04ff1a533119d589baee777c21194a951168b0c7"},"cell_type":"markdown","source":"
\n## 6- EDA\n In this section, you'll learn how to use graphical and numerical techniques to begin uncovering the structure of your data. \n \n* Which variables suggest interesting relationships?\n* Which observations are unusual?\n* Analysis of the features!\n\nBy the end of the section, you'll be able to answer these questions and more, while generating graphics that are both insightful and beautiful. then We will review analytical and statistical operations:\n\n1. Data Collection\n1. Visualization\n1. Data Cleaning\n1. Data Preprocessing\n\n\n ###### [Go to top](#top)"},{"metadata":{"_uuid":"cedecea930b278f86292367cc28d2996a235a169"},"cell_type":"markdown","source":"
\n## 6-1 Data Collection\n**Data collection** is the process of gathering and measuring data, information or any variables of interest in a standardized and established manner that enables the collector to answer or test hypothesis and evaluate outcomes of the particular collection.[techopedia]\n\nI start Collection Data by the training and testing datasets into **Pandas DataFrames**.\n###### [Go to top](#top)"},{"metadata":{"_kg_hide-input":true,"_uuid":"9269ae851b744856bce56840637030a16a5877e1","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58ed9c838069f54de5cf90b20a774c3e236149b3"},"cell_type":"markdown","source":"**<< Note 1 >>**\n\n* Each **row** is an observation (also known as : sample, example, instance, record).\n* Each **column** is a feature (also known as: Predictor, attribute, Independent Variable, input, regressor, Covariate).\n###### [Go to top](#top)"},{"metadata":{"_uuid":"4708d70e39d1ae861bbf34411cf03d07f261fceb","trusted":true},"cell_type":"code","source":"train.sample(1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8e7a84ab982504d7263b1812fa66bba78bddbdc","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test.sample(1) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3483fbc1e932d9f387703a796248963e77cefa1d"},"cell_type":"markdown","source":"Or you can use others command to explorer dataset, such as "},{"metadata":{"_uuid":"08a94b16129d4c231b64d4691374e18aa80f1d80","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.tail(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"581b90e6a869c3793472c7edd59091d6d6342fb2"},"cell_type":"markdown","source":"
\n## 6-1-1 Features\nFeatures can be from following types:\n* numeric\n* categorical\n* ordinal\n* datetime\n* coordinates\n\nFind the type of features in **Qoura dataset**?!\n\nFor getting some information about the dataset you can use **info()** command."},{"metadata":{"_kg_hide-input":true,"_uuid":"ca840f02925751186f87e402fcb5f637ab1ab8a0","trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cbcf76344a6e3c8e841ccf1f43bf00d040a06a1","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73ab30f86273b590a51fc363d9bf78c2709558fa"},"cell_type":"markdown","source":"
\n## 6-1-2 Explorer Dataset\n1- Dimensions of the dataset.\n\n2- Peek at the data itself.\n\n3- Statistical summary of all attributes.\n\n4- Breakdown of the data by the class variable.\n\nDon’t worry, each look at the data is **one command**. These are useful commands that you can use again and again on future projects.\n###### [Go to top](#top)"},{"metadata":{"_kg_hide-input":true,"_uuid":"4b45251be7be77333051fe738639104ae1005fa5","trusted":true},"cell_type":"code","source":"# shape for train and test\nprint('Shape of train:',train.shape)\nprint('Shape of test:',test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"c64e9d3e0bf394fb833de94a0fc5c34f69fce24c","trusted":true},"cell_type":"code","source":"#columns*rows\ntrain.size","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7b5fd1034cd591ebd29fba1c77d342ec2b408d13"},"cell_type":"markdown","source":"After loading the data via **pandas**, we should checkout what the content is, description and via the following:"},{"metadata":{"_kg_hide-input":true,"_uuid":"edd043f8feb76cfe51b79785302ca4936ceb7b51","trusted":true},"cell_type":"code","source":"type(train)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"edd043f8feb76cfe51b79785302ca4936ceb7b51","trusted":true},"cell_type":"code","source":"type(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b8b6f0c962a59e5258e74ed9e740a4aaf7c8113","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c288c3dc8656a872a8529368812546e434d3a22"},"cell_type":"markdown","source":"To pop up 5 random rows from the data set, we can use **sample(5)** function and find the type of features."},{"metadata":{"_uuid":"09eb18d1fcf4a2b73ba2f5ddce99dfa521681140","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.sample(5) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8280749a19af32869978c61941d1dea306632d71"},"cell_type":"markdown","source":"
\n## 6-2 Data Cleaning\nWhen dealing with real-world data, dirty data is the norm rather than the exception. We continuously need to predict correct values, impute missing ones, and find links between various data artefacts such as schemas and records. We need to stop treating data cleaning as a piecemeal exercise (resolving different types of errors in isolation), and instead leverage all signals and resources (such as constraints, available statistics, and dictionaries) to accurately predict corrective actions.\n\nThe primary goal of data cleaning is to detect and remove errors and **anomalies** to increase the value of data in analytics and decision making. While it has been the focus of many researchers for several years, individual problems have been addressed separately. These include missing value imputation, outliers detection, transformations, integrity constraints violations detection and repair, consistent query answering, deduplication, and many other related problems such as profiling and constraints mining.[4]\n###### [Go to top](#top)"},{"metadata":{"_uuid":"a6315bf510cecb907b2d23aad25faf6ccad32ac4"},"cell_type":"markdown","source":"How many NA elements in every column!!\n\nGood news, it is Zero!\n\nTo check out how many null info are on the dataset, we can use **isnull().sum()**."},{"metadata":{"_kg_hide-input":true,"_uuid":"675f72fb58d83c527f71819e71ed8e17f81126f5","trusted":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5faa6528c6667060c05268757ff46e211b4fea3f"},"cell_type":"markdown","source":"But if we had , we can just use **dropna()**(be careful sometimes you should not do this!)"},{"metadata":{"_kg_hide-input":true,"_uuid":"e8e124ca20643ad307d9bfdc34328d548c6ddcbc","trusted":true},"cell_type":"code","source":"# remove rows that have NA's\nprint('Before Droping',train.shape)\ntrain = train.dropna()\nprint('After Droping',train.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"277e1998627d6a3ddeff4e913a6b8c3dc81dec96"},"cell_type":"markdown","source":"\nWe can get a quick idea of how many instances (rows) and how many attributes (columns) the data contains with the shape property."},{"metadata":{"_uuid":"c2f1eaf0b6dfdc7cc4dace04614e99ed56425d00"},"cell_type":"markdown","source":"To print dataset **columns**, we can use columns atribute."},{"metadata":{"_uuid":"909d61b33ec06249d0842e6115597bbacf21163f","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3458838205be4c7fbff88e95ef69934e13e2199b"},"cell_type":"markdown","source":"You see number of unique item for Target with command below:"},{"metadata":{"_kg_hide-input":true,"_uuid":"c7937700664991b29bdb0b3f04942c59498da760","trusted":true},"cell_type":"code","source":"train_target = train['target'].values\n\nnp.unique(train_target)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d824cb29e135dc5ae98964e71ec0adc0e05ebd43"},"cell_type":"markdown","source":"YES, quora problem is a **binary classification**! :)"},{"metadata":{"_uuid":"ae08b544a8d4202c7d0a47ec83d685e81c91a66d"},"cell_type":"markdown","source":"To check the first 5 rows of the data set, we can use head(5)."},{"metadata":{"_kg_hide-input":true,"_uuid":"5899889553c3416b27e93efceddb106eb71f5156","trusted":true},"cell_type":"code","source":"train.head(5) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1150b6ac3d82562aefd5c64f9f01accee5eace4d"},"cell_type":"markdown","source":"Or to check out last 5 row of the data set, we use tail() function."},{"metadata":{"_uuid":"79339442ff1f53ae1054d794337b9541295d3305","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.tail() ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8a1cc36348c68fb98d6cb28aa9919fc5f2892f3"},"cell_type":"markdown","source":"To give a **statistical summary** about the dataset, we can use **describe()**\n"},{"metadata":{"_uuid":"3f7211e96627b9a81c5b620a9ba61446f7719ea3","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.describe() ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10bdb8246f66c14043392806cae714f688cc8251"},"cell_type":"markdown","source":"As you can see, the statistical information that this command gives us is not suitable for this type of data\n**describe() is more useful for numerical data sets**"},{"metadata":{"_uuid":"91dda1f631cf4ed362162501aaaac6d19cfd6cc7"},"cell_type":"markdown","source":"
\n## 6-3 Data Preprocessing\n**Data preprocessing** refers to the transformations applied to our data before feeding it to the algorithm.\n \nData Preprocessing is a technique that is used to convert the raw data into a clean data set. In other words, whenever the data is gathered from different sources it is collected in raw format which is not feasible for the analysis.\nthere are plenty of steps for data preprocessing and we just listed some of them in general(Not just for Quora) :\n1. removing Target column (id)\n1. Sampling (without replacement)\n1. Making part of iris unbalanced and balancing (with undersampling and SMOTE)\n1. Introducing missing values and treating them (replacing by average values)\n1. Noise filtering\n1. Data discretization\n1. Normalization and standardization\n1. PCA analysis\n1. Feature selection (filter, embedded, wrapper)\n1. Etc.\n\nWhat methods of preprocessing can we run on Quora?! \n###### [Go to top](#top)"},{"metadata":{"_uuid":"6c8c838f497c66a227975fb9a2f588e431f0c568"},"cell_type":"markdown","source":"**<< Note 2 >>**\nin pandas's data frame you can perform some query such as \"where\""},{"metadata":{"_uuid":"c8c8d9fd63d9bdb601183aeb4f1435affeb8a596","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.where(train ['target']==1).count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"33fc33a18489b438a884819d99dc00a02b113be8"},"cell_type":"markdown","source":"As you can see in the below in python, it is so easy perform some query on the dataframe:"},{"metadata":{"_uuid":"8b545ff7e8367c5ab9c1db710f70b6936ac8422c","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train[train['target']>1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2788d023986eca622f7db9e1d64c2a4e02737ddb"},"cell_type":"markdown","source":"Some examples of questions that they are insincere"},{"metadata":{"_uuid":"d517b2b99a455a6b89c238faf1647515b8a67d87","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train[train['target']==1].head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b67d109a0cec1a5475b863bbce8aa3ac9d2d4fb"},"cell_type":"markdown","source":"
\n## 6-3-1 Is data set imbalance?\n"},{"metadata":{"_uuid":"4218d492753322c50142021833efb24cfdfc6ad3","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_target.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e058f90ca403f00d91d0405a7d8822dc7d6de55"},"cell_type":"markdown","source":"A large part of the data is unbalanced, but **how can we solve it?**"},{"metadata":{"_kg_hide-input":true,"_uuid":"dc6340ee1b637d192e29cbc8d3744ae6351b9c8b","trusted":true},"cell_type":"code","source":"train[\"target\"].value_counts()\n# data is imbalance","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c948aad0aefbea90524b91c771ad74b6261387c"},"cell_type":"markdown","source":"**Imbalanced dataset** is relevant primarily in the context of supervised machine learning involving two or more classes. \n\n**Imbalance** means that the number of data points available for different classes is different:\nIf there are two classes, then balanced data would mean 50% points for each of the class. For most machine learning techniques, little imbalance is not a problem. So, if there are 60% points for one class and 40% for the other class, it should not cause any significant performance degradation. Only when the class imbalance is high, e.g. 90% points for one class and 10% for the other, standard optimization criteria or performance measures may not be as effective and would need modification.\n\n\n\n[Image source](http://api.ning.com/files/vvHEZw33BGqEUW8aBYm4epYJWOfSeUBPVQAsgz7aWaNe0pmDBsjgggBxsyq*8VU1FdBshuTDdL2-bp2ALs0E-0kpCV5kVdwu/imbdata.png)\n\nA typical example of imbalanced data is encountered in e-mail classification problem where emails are classified into ham or spam. The number of spam emails is usually lower than the number of relevant (ham) emails. So, using the original distribution of two classes leads to imbalanced dataset.\n\nUsing accuracy as a performace measure for highly imbalanced datasets is not a good idea. For example, if 90% points belong to the true class in a binary classification problem, a default prediction is true for all data poimts leads to a classifier which is 90% accurate, even though the classifier has not learnt anything about the classification problem at hand![9]"},{"metadata":{"_uuid":"873b2cfdee04b8ba087df1c4bf01ae69ef2f1c52"},"cell_type":"markdown","source":"
\n## 6-3-2 Exploring the question"},{"metadata":{"_kg_hide-input":true,"_uuid":"e445c859d7c43857cfbf370ff20060a5341d3c89","trusted":true},"cell_type":"code","source":"question = train['question_text']\ni=0\nfor q in question[:5]:\n i=i+1\n print('sample '+str(i)+':' ,q)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"fa78f61df85a9c76bd092dbf6d6bcec4b6b2631f","trusted":true},"cell_type":"code","source":"text_withnumber = train['question_text']\nresult = ''.join([i for i in text_withnumber if not i.isdigit()])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c50c6c2683c5c08a6c9c34b75be61567d5993fa0"},"cell_type":"markdown","source":"
\n## 6-3-2 Some Feature Engineering"},{"metadata":{"_uuid":"c641754f26e07c368596af3054268f1b3b764921"},"cell_type":"markdown","source":"[NLTK](https://www.nltk.org/) is one of the leading platforms for working with human language data and Python, the module NLTK is used for natural language processing. NLTK is literally an acronym for Natural Language Toolkit.\n\nWe get a set of **English stop** words using the line"},{"metadata":{"_kg_hide-input":true,"_uuid":"10ca7d56255b95fc774fff5adf7b4273ec7a1ea2","trusted":true},"cell_type":"code","source":"#from nltk.corpus import stopwords\neng_stopwords = set(stopwords.words(\"english\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f5af107041b279ce723761f37f4ffebae2b22a3"},"cell_type":"markdown","source":"The returned list stopWords contains **179 stop words** on my computer.\nYou can view the length or contents of this array with the lines:"},{"metadata":{"_kg_hide-input":true,"_uuid":"eca2d53bfae70c55b3b5b0e2c244826465cb478b","trusted":true},"cell_type":"code","source":"print(len(eng_stopwords))\nprint(eng_stopwords)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f049c6d9633200496ed97f8066257849b4824da"},"cell_type":"markdown","source":"The metafeatures that we'll create based on SRK's EDAs, [sudalairajkumar](http://http://www.kaggle.com/sudalairajkumar/simple-feature-engg-notebook-spooky-author) and [tunguz](https://www.kaggle.com/tunguz/just-some-simple-eda) are:\n1. Number of words in the text\n1. Number of unique words in the text\n1. Number of characters in the text\n1. Number of stopwords\n1. Number of punctuations\n1. Number of upper case words\n1. Number of title case words\n1. Average length of the words\n\n###### [Go to top](#top)"},{"metadata":{"_uuid":"f4982fc699bcb147513c247b9f4d86b02902eded"},"cell_type":"markdown","source":"Number of words in the text "},{"metadata":{"_kg_hide-input":true,"_uuid":"5b29fbd86ab48be6bd84fcac6fb6bca84d4b8792","trusted":true},"cell_type":"code","source":"train[\"num_words\"] = train[\"question_text\"].apply(lambda x: len(str(x).split()))\ntest[\"num_words\"] = test[\"question_text\"].apply(lambda x: len(str(x).split()))\nprint('maximum of num_words in train',train[\"num_words\"].max())\nprint('min of num_words in train',train[\"num_words\"].min())\nprint(\"maximum of num_words in test\",test[\"num_words\"].max())\nprint('min of num_words in train',test[\"num_words\"].min())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"83becd8affc2abab2252e065f77c80dc0dcf53be"},"cell_type":"markdown","source":"Number of unique words in the text"},{"metadata":{"_kg_hide-input":true,"_uuid":"72aebb943122982b891c959fa9fa36224adcb2fc","trusted":true},"cell_type":"code","source":"train[\"num_unique_words\"] = train[\"question_text\"].apply(lambda x: len(set(str(x).split())))\ntest[\"num_unique_words\"] = test[\"question_text\"].apply(lambda x: len(set(str(x).split())))\nprint('maximum of num_unique_words in train',train[\"num_unique_words\"].max())\nprint('mean of num_unique_words in train',train[\"num_unique_words\"].mean())\nprint(\"maximum of num_unique_words in test\",test[\"num_unique_words\"].max())\nprint('mean of num_unique_words in train',test[\"num_unique_words\"].mean())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb2719fa417f2c3fabea9b6582081738ecdf678b"},"cell_type":"markdown","source":"Number of characters in the text "},{"metadata":{"_kg_hide-input":true,"_uuid":"a7029af9cfed9eb2e624d7177887e111a71054ff","trusted":true},"cell_type":"code","source":"train[\"num_chars\"] = train[\"question_text\"].apply(lambda x: len(str(x)))\ntest[\"num_chars\"] = test[\"question_text\"].apply(lambda x: len(str(x)))\nprint('maximum of num_chars in train',train[\"num_chars\"].max())\nprint(\"maximum of num_chars in test\",test[\"num_chars\"].max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ddd289db2420b4f3fee7268fef94926688afd203"},"cell_type":"markdown","source":"Number of stopwords in the text"},{"metadata":{"_kg_hide-input":true,"_uuid":"086e229b918087420c33b57c7ad51d6723cf70f7","trusted":true},"cell_type":"code","source":"train[\"num_stopwords\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).lower().split() if w in eng_stopwords]))\ntest[\"num_stopwords\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).lower().split() if w in eng_stopwords]))\nprint('maximum of num_stopwords in train',train[\"num_stopwords\"].max())\nprint(\"maximum of num_stopwords in test\",test[\"num_stopwords\"].max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"428a009f1a3b00b73ec7d6e8558aebc995e42594"},"cell_type":"markdown","source":"Number of punctuations in the text"},{"metadata":{"_kg_hide-input":true,"_uuid":"947abd63c51d74dc33c2891fb1e1b9381d9da23c","trusted":true},"cell_type":"code","source":"train[\"num_punctuations\"] =train['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]) )\ntest[\"num_punctuations\"] =test['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]) )\nprint('maximum of num_punctuations in train',train[\"num_punctuations\"].max())\nprint(\"maximum of num_punctuations in test\",test[\"num_punctuations\"].max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a93e8fd32c2ff7dffd81f62ff3b6b3a5975d6836"},"cell_type":"markdown","source":"Number of title case words in the text"},{"metadata":{"_kg_hide-input":true,"_uuid":"82c95fcf5848ca383a6a84501fe74fef371392d1","trusted":true},"cell_type":"code","source":"train[\"num_words_upper\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\ntest[\"num_words_upper\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\nprint('maximum of num_words_upper in train',train[\"num_words_upper\"].max())\nprint(\"maximum of num_words_upper in test\",test[\"num_words_upper\"].max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bc4c2b642cb8adf12fd6dbb01616079a454d384"},"cell_type":"markdown","source":"Number of title case words in the text"},{"metadata":{"_kg_hide-input":true,"_uuid":"b938bdcfe7c418f5b4d57c9fd21c77d8bf4d3f06","trusted":true},"cell_type":"code","source":"train[\"num_words_title\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\ntest[\"num_words_title\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\nprint('maximum of num_words_title in train',train[\"num_words_title\"].max())\nprint(\"maximum of num_words_title in test\",test[\"num_words_title\"].max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c81bb79d34d242f74b8ed650a8998efdb29e38b"},"cell_type":"markdown","source":" Average length of the words in the text "},{"metadata":{"_kg_hide-input":true,"_uuid":"3058236ff8702754ee4132f7eb705dd54f354af4","trusted":true},"cell_type":"code","source":"train[\"mean_word_len\"] = train[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))\ntest[\"mean_word_len\"] = test[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))\nprint('mean_word_len in train',train[\"mean_word_len\"].max())\nprint(\"mean_word_len in test\",test[\"mean_word_len\"].max())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c91162602814ba230ab9fe30f9941ac6409133b9"},"cell_type":"markdown","source":"We add some new feature to train and test data set now, print columns agains"},{"metadata":{"_uuid":"05cae032149a7c79a92a3b2bf80185c483d0e976","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(train.columns)\ntrain.head(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa882e5bcdc7d5f440489eff75d1d225269655a4"},"cell_type":"markdown","source":"**<< Note >>**\n>**Preprocessing and generation pipelines depend on a model type**"},{"metadata":{"_uuid":"f453f5a76194116a73ce8ae5c98de980dc8b5758"},"cell_type":"markdown","source":"## What is Tokenizer?\nTokenizing raw text data is an important pre-processing step for many NLP methods. As explained on **wikipedia**, tokenization is “the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.” In the context of actually working through an NLP analysis, this usually translates to converting a string like \"My favorite color is blue\" to a list or array like [\"My\", \"favorite\", \"color\", \"is\", \"blue\"].[11]"},{"metadata":{"trusted":true,"_uuid":"b25495da861c8f917ff91c5c9296b954de03983f"},"cell_type":"code","source":"import nltk\nmystring = \"I love Kaggle\"\nmystring2 = \"I'd love to participate in kaggle competitions.\"\nnltk.word_tokenize(mystring)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb760f3b6a02e2b4d5f371bb1fb5376f1cf7db88"},"cell_type":"code","source":"nltk.word_tokenize(mystring2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"055772bd170aa8018aabd85106b76675802c33b3"},"cell_type":"markdown","source":"
\n## 6-4 Data Visualization\n**Data visualization** is the presentation of data in a pictorial or graphical format. It enables decision makers to see analytics presented visually, so they can grasp difficult concepts or identify new patterns.\n\n> * Two** important rules** for Data visualization:\n> 1. Do not put too little information\n> 1. Do not put too much information\n\n###### [Go to top](#top)"},{"metadata":{"_uuid":"5d991f5a4a9e4fffcbcee4a51b3cf1cd95007427"},"cell_type":"markdown","source":"
\n## 6-4-1 CountPlot"},{"metadata":{"_kg_hide-input":true,"_uuid":"1b54931579ed4e3004369a59fe9c6f23b97719de","trusted":true},"cell_type":"code","source":"ax=sns.countplot(x='target',hue=\"target\", data=train ,linewidth=5,edgecolor=sns.color_palette(\"dark\", 3))\nplt.title('Is data set imbalance?');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"eb15e8fed181179a086bf0db7dc21eaabb4eb088","trusted":true},"cell_type":"code","source":"ax = sns.countplot(y=\"target\", hue=\"target\", data=train)\nplt.title('Is data set imbalance?');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be2b936baaa6dcc2d574a861c75584ed04d3589e"},"cell_type":"markdown","source":"
\n## 6-4-2 Pie Plot"},{"metadata":{"_kg_hide-input":true,"_uuid":"4a2332f8c87da0a4f8cc31f587ce547470a0d615","trusted":true},"cell_type":"code","source":"\nax=train['target'].value_counts().plot.pie(explode=[0,0.1],autopct='%1.1f%%' ,shadow=True)\nax.set_title('target')\nax.set_ylabel('')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08745636a4aef9797daf0f52610cdd84d6cfd8f7"},"cell_type":"markdown","source":"
\n## 6-4-6 WordCloud"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}