{"cells":[{"metadata":{"_uuid":"bc0c482336814e6795e18ce784b011bfd88dbba1"},"cell_type":"markdown","source":"

Quora - Detecting Insincere Questions

"},{"metadata":{"_uuid":"49078f1f1538bfb3d63202f052e97763452db54e"},"cell_type":"markdown","source":"

Dejan Porjazovski and Donayorov Maksad

"},{"metadata":{"_uuid":"05ee2be39946c97dc08f9e16b4a48fc0c250fdb4"},"cell_type":"markdown","source":"# Introduction\n\nThis notebook aims to show an approach on how to solve an interesting problem, which resolves around classifying insincere questions from the famous platform Quora. The task is a binary classification problem. Such tasks are implemented in many applications that we use in our everyday lives, for example email spam detection, comments with profane language and etc. In the report, we start with a focus on exploring the data set and after obtaining information about it, we preprocess it in order to feed it to a classifier. We use the FastText classifier developed by Facebook. At the end we evaluate the performance of our classifier by submitting it's predictions results to Kaggle's \"Quora Insincere Questions Classification\" competition. The best score that we got using the techniques stated in this report is 0.573 and the leader of the competition has a score of 0.729 (at the time when this report was written).\n\nThe aim of this project is to provide an introductory step by step clarification on how a real-world text classification problem could be resolved by analyzing the data, using common tools for preprocessing and getting a reasonable accuracy score. \n\nSome the questions that we will try to address, are:\n\n"},{"metadata":{"_uuid":"2365127a67fe9d491a88cf89c026aaa17ca5e762"},"cell_type":"markdown","source":"# Data description\n\nThe data that is provided consist of train data, test data and word embedding. Due to some limitations we are not going to use the provided embedding.\n\nThe train data consists of 1,306,122 samples and has three features: *qid*, *question_text* and *target*. \n
\nThe number of positive questions is 1,225,312, while the number of insincere is 80,810. This is an indication of a highly imbalanced data set.\n
\nThe test data consists of 56,370 samples and has two features: *qid* and *question_text*."},{"metadata":{"_uuid":"2b5c133ba2dab86bb63bb25434fc43004ae29e1f"},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_uuid":"e5894c742e2f9902534915552adfe628551197c9"},"cell_type":"markdown","source":"Let's start by importing the main Python packages that will be used throughout this notebook:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Libraries\nimport numpy as np \nimport pandas as pd\nimport os\nimport random\nimport re\nfrom pathlib import Path\nimport fastText as ft\nimport pandas as pd\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import accuracy_score\nfrom collections import Counter\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"011731a7d6c369aa21039e0c2487f3f531e0daf4"},"cell_type":"markdown","source":"Next, by using Python package *pandas* we can import the testing and training data. "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# import data\ndf_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be9098796a5b0444ffc022c0344c2bd1ba6d8781"},"cell_type":"markdown","source":"The rule of thumb is to have a quick look into the data, which gives an insight about we are dealing with."},{"metadata":{"trusted":true,"_uuid":"f499f150f0a9cecf510bdad3cfc4d558969f3577"},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"287a205b4513836af51dc37bd4292505eacb8b15"},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7bddf674d7bfe95b05a71d97ca4c0309503e4f4"},"cell_type":"markdown","source":"Once we have seen how the data looks like, it is a good idea to the distribution of labeled data. *0* means all the question that are sincere and *1* are questions tha aren't."},{"metadata":{"trusted":true,"_uuid":"9cb7ce25dd4a0693c909fcb3f0d9c519f36e5bd7"},"cell_type":"code","source":"train_values = df_train['target'].values\nzeros = np.where(train_values == 0)\n\ntrain_values = df_train['target'].values\nones = np.where(train_values == 1)\n\ny = [len(zeros[0]), len(ones[0])]\nx = [0, 1]\n\nplt.bar([0, 1], y)\nplt.xticks(x, (0, 1))\nplt.title('Class distribution')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b7031b2d0757d006ca52fd40478f221601b58fd9"},"cell_type":"markdown","source":"From the above histogram we can see that the ration of *0*s is much higher then the ratio of *1*s. This makes the classification task more complicated, as a trained model could become more bias to the dominant class, which in this case is *0* or sincere questions and predicting the label for *1* becomes harder as the data is very unbalanced."},{"metadata":{"_uuid":"b56d5b592614baed6a71b31afff930811aa8b311"},"cell_type":"markdown","source":"# Preprocessing"},{"metadata":{"_uuid":"55b048dcd87e88ca51bc5b8f0fe917eae6f68840"},"cell_type":"markdown","source":"Let's start by importing necessary packages for the preprocessing of the questions text."},{"metadata":{"trusted":true,"_uuid":"00378571aa78064785ef2000ba3baf4223b53efb"},"cell_type":"code","source":"import re, string, unicodedata\nimport nltk\nfrom nltk import word_tokenize, sent_tokenize\nfrom nltk.corpus import stopwords\nfrom nltk.stem import LancasterStemmer, WordNetLemmatizer","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37b260fa432a1fecd4a766e076da120c64fbc162"},"cell_type":"markdown","source":"It is very important that cleanup the data, as the questions text might include lots of words that add noise to the data and the classifier might have hard time to predict the true label. For that reason we will normalize the question text by removing all the non ASCII characters, converting the strings into lower case, removing punctuation, converting the words into stem words and lemmatize the words so they all have a unified format. Once all of these steps are preformed we return a normalized string. For example:\n\n```\nHow did Quebec nationalists see their province as a nation in the 1960s?\nDo you have an adopted dog, how would you encourage people to adopt and not shop?\nHow did Otto von Guericke used the Magdeburg hemispheres?\nCan I convert montra helicon D to a mountain bike by just changing the tyres?\n```\n\nbecomes:\n```\nhow do quebec nationalists see their province as a nation in the 1960s\ndo you have an adopt dog how would you encourage people to adopt and not shop\nhow do otto von guericke use the magdeburg hemispheres\ncan i convert montra helicon d to a mountain bike by just change the tyres\n```\n\nEven though removing stop words is a common practice, we decided to not do it on our data set as we have noticed that it decreased our score."},{"metadata":{"trusted":true,"_uuid":"c01674f83001f9e43f9713f36f661c13facb7757"},"cell_type":"code","source":"stemmer = LancasterStemmer()\nlemmatizer = WordNetLemmatizer()\n\ndef remove_non_ascii(word):\n \"\"\"Remove non-ASCII characters from list of tokenized words\"\"\"\n new_word = unicodedata.normalize('NFKD', word).encode('ascii', 'ignore').decode('utf-8', 'ignore')\n return new_word\n\ndef to_lowercase(word):\n \"\"\"Convert all characters to lowercase from list of tokenized words\"\"\"\n return word.lower()\n\ndef remove_punctuation(word):\n \"\"\"Remove punctuation from list of tokenized words\"\"\"\n new_word = re.sub(r'[^\\w\\s]', '', word)\n return new_word\n\ndef remove_stopwords(word):\n \"\"\"Remove stop words from list of tokenized words\"\"\"\n if word not in stopwords.words('english'):\n return word\n return ''\n\ndef stem_words(word):\n \"\"\"Stem words in list of tokenized words\"\"\"\n return stemmer.stem(word)\n\ndef lemmatize_verbs(word):\n \"\"\"Lemmatize verbs in list of tokenized words\"\"\"\n return lemmatizer.lemmatize(word, pos='v')\n\ndef normalize(word):\n word = remove_non_ascii(word)\n word = to_lowercase(word)\n word = remove_punctuation(word)\n # word = remove_stopwords(word)\n word = lemmatize_verbs(word)\n return word\n\ndef get_processed_text(string):\n words = nltk.word_tokenize(string)\n new_words = []\n for word in words:\n new_word = normalize(word)\n if new_word != '':\n new_words.append(new_word)\n return ' '.join(new_words)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c3ee563556d6aaac1266313145b6a5439f2480e1"},"cell_type":"markdown","source":"Now that our `get_processed_text` function is ready we can convert all the rows in the `question_text` colum of the `df_train` data frame into \"clean\" version:"},{"metadata":{"trusted":true,"_uuid":"0941ced5c0709c394ab8290178afd90a5988429b"},"cell_type":"code","source":"df_train.question_text = df_train.question_text.apply(lambda x: get_processed_text(x))\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3134142ea54eb140d57882685169a2ec551319d1"},"cell_type":"markdown","source":"Next, we need to create a `label_and_text` column in the `df_train` dataframe. This column will have rows of concatenated label and text that will be used later when we train our classifier using FastText."},{"metadata":{"trusted":true,"_uuid":"939cc87e9525595b0f1d7fe1e9f324a56aadbaa3"},"cell_type":"code","source":"df_train['label_and_text'] = '__label__' + df_train.target.map(str) + ' '+ df_train.question_text\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1df6348468045ef017a22650a0bbc4de01797138"},"cell_type":"markdown","source":"Same preprocessing should be done on the test data, except we don't have to add any extra column."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"116cc7c17d5b0aae94437bbaf792e4585923652b"},"cell_type":"code","source":"df_test.question_text = df_test.question_text.apply(lambda x: get_processed_text(x))\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2c8540feab19500e93a6592d9ce2e5b6a1a31e5b"},"cell_type":"markdown","source":"FastText trains a classifier with a training data and it receives as a parameter the path to that training data file. For that reason we have to create a `train.txt` file and export all of the rows in the column `label_and_text`:"},{"metadata":{"trusted":true,"_uuid":"812187b769b74cdf532846985d0a0757506e3902"},"cell_type":"code","source":"# Write training data to a file as required by fasttext\ntraining_file = open('train.txt','w')\ntraining_file.writelines(df_train.label_and_text + '\\n')\ntraining_file.close()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f7c48edffae613a65f194550dbc6f9f2cb40d29"},"cell_type":"markdown","source":"# FastText\n\nFastText is an open-souce, lightweight library, developed by Facebook and it is used for efficient learning of word representations and sentence classification. \n
\nThe model allows to create an unsupervised learning or supervised learning algorithm for obtaining vector representations for word. It uses a neural network for creating the word embeddings."},{"metadata":{"_uuid":"766514ba016284ab2d0cf70e5fe10063478d7543"},"cell_type":"markdown","source":"# Model tuning \n\nThe next chunk of code tries to find the best hyperparameters for the FastText algorithm. Mainly, we are interested in `lr`,`wordNgrams` ,`epoch` ,`ws` ,`loss`. Here is an explanation for each of them with their default values:\n\n```\nwordNgrams: max length of word ngram [1]\nlr: learning rate [0.05]\nepoch: number of epochs [5]\nws: size of the context window [5]\nloss: loss function {ns, hs, softmax} [ns]\n```\n\nThe default values should be good enough for many classification problems, but unfortunately in our case we have to perform a grid search and find the best hyperparameter that classifies our data the best. The reason that it is mandatory in this case is that our data is very unbalanced. We will be looping though these values:\n\n```\nlr = [0.05, 0.1, 0.2],\nwordNgrams = [1, 2, 3],\nepoch = [1, 5],\nws = [5, 10],\nloss = ['ns', 'hs', 'softmax']\n```\n\nAs you might have noticed there are 108 iterations:\n$3 * 3 * 2 * 2 * 3 = 3^3 * 2^2 =108$ \n\nOn avarage one itiration can take up to 2 minutes that means that the grid search will take about 3.6 hours: $108*2mins/60mins = 3.6hours$. For that reason we have to stick to these parameters and certainly if we notice some unusual phenomenon, then we will perform another grid search with the best results that these parameters give us.\n\nWe would like to thank Randi Griffin for the post: *\"Facebook's fastText algorithm\"* as the some of the implementations are based on that post."},{"metadata":{"trusted":true,"_uuid":"fb3c546541d87efc91576f5735480ad4d616f7cc"},"cell_type":"code","source":"# Function to do K-fold CV across different fasttext parameter values\ndef tune(Y, X, YX, k, lr, wordNgrams, epoch, loss, ws):\n # Record results\n results = []\n for lr_val in lr:\n for wordNgrams_val in wordNgrams:\n for epoch_val in epoch: \n for loss_val in loss:\n for ws_val in ws:\n # K-fold CV\n kf = KFold(n_splits=k, shuffle=True)\n fold_results = []\n # For each fold\n for train_index, test_index in kf.split(X):\n # Write the training data for this CV fold\n training_file = open('train_cv.txt','w')\n training_file.writelines(YX[train_index] + '\\n')\n training_file.close()\n # Fit model for this set of parameter values\n model = ft.FastText.train_supervised(\n 'train_cv.txt',\n lr=lr_val,\n wordNgrams=wordNgrams_val,\n epoch=epoch_val,\n loss=loss_val,\n ws=ws_val\n )\n # Predict the holdout sample\n pred = model.predict(X[test_index].tolist())\n pred = pd.Series(pred[0]).apply(lambda x: int(re.sub('__label__', '', x[0])))\n # Compute accuracy for this CV fold\n fold_results.append(accuracy_score(Y[test_index], pred.values))\n # Compute mean accuracy across 10 folds \n mean_acc = pd.Series(fold_results).mean()\n print([lr_val, wordNgrams_val, epoch_val, loss_val, ws_val, mean_acc])\n # Add current parameter values and mean accuracy to results table\n results.append([lr_val, wordNgrams_val, epoch_val, loss_val, ws_val, mean_acc]) \n # Return as a DataFrame \n results = pd.DataFrame(results)\n results.columns = ['lr','wordNgrams','epoch','loss','ws_val','mean_acc']\n return(results)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed2ed83c2eacd08909d9a4c0bf063b8993ae9b42"},"cell_type":"markdown","source":"Now that our `tune` function is ready we have to perform an exhaustive search."},{"metadata":{"trusted":true,"_uuid":"fcb4b59d2d6e8a169d7ce68518eaa465f9611e38","scrolled":false},"cell_type":"code","source":"# results = tune(\n# Y = df_train.target,\n# X = df_train.question_text,\n# YX = df_train.label_and_text,\n# k = 5, \n# lr = [0.05, 0.1, 0.2],\n# wordNgrams = [1, 2, 3],\n# epoch = [1, 5],\n# ws=[5, 10],\n# loss=['ns', 'hs', 'softmax']\n# )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f00ac7be30ed035ce3054898413dabc1ea0a325"},"cell_type":"markdown","source":"*We will comment the content of this box, as we don't want it to be executed during the committing.*"},{"metadata":{"_uuid":"a908721ed250ccdd231ca35b65ad77799ed3690e"},"cell_type":"markdown","source":"This function call will have 108 iteration and print the best hyperparameters that work on our data set. The output looks like:\n\n```\nlr wordNgrams epoch loss ws_val mean_acc\n-----------------------------------------------------------\n[0.05, 1, 1, 'ns', 5, 0.951719]\n[0.05, 1, 1, 'ns', 10, 0.951705]\n[0.05, 1, 1, 'hs', 5, 0.951468]\n[0.05, 1, 1, 'hs', 10, 0.951427]\n[0.05, 1, 1, 'softmax', 5, 0.951685]\n...\n[0.1, 3, 5, 'hs', 10, 0.954775]\n[0.1, 3, 5, 'softmax', 5, 0.954365]\n[0.1, 3, 5, 'softmax', 10, 0.954459]\n[0.2, 1, 1, 'ns', 5, 0.951787]\n[0.2, 1, 1, 'ns', 10, 0.951823]\n[0.2, 1, 1, 'hs', 5, 0.951860]\n```\n\nFrom which we will choose the ones with the highest accuracy and do our predictions."},{"metadata":{"trusted":true,"_uuid":"4b916b983bff3b55f5a09f3fe647a897451e865c"},"cell_type":"markdown","source":"From the entire 180 outputs we noticed that our top 10 hyperparameters are:\n```\n lr wordNgrams epoch loss ws_val mean_acc\n ----------------------------------------------------------------\n0.95502 [0.05, 2, 5, 'softmax', 5, 0.9550279373942934]\n0.95507 [0.05, 3, 5, 'softmax', 5, 0.9550777033236294]\n0.95510 [0.05, 2, 5, 'hs', 5, 0.9551098597511588]\n0.95511 [0.05, 2, 5, 'softmax', 10, 0.9551198128567192]\n0.95512 [0.05, 3, 5, 'softmax', 10, 0.9551282348207829]\n0.95513 [0.05, 2, 5, 'hs', 10, 0.9551343592226385]\n0.95515 [0.05, 3, 5, 'hs', 10, 0.9551504378144902]\n0.95539 [0.05, 3, 5, 'hs', 5, 0.9553962030503383]\n```\n\nFrom this we can immoderately notice that:\n1. A smaller value of `lr` is better\n2. `wordNgrams` > 1 work better\n3. The `epoch` could be increased\n4. The `softmax` and `hs` wok better as a value of`loss`\n5. `ws` value can be increased \n\nWih these in mind we can perform our next grid search."},{"metadata":{"trusted":true,"_uuid":"83d6f9adc6ec98cf5add0d2ce8380dfcab3461d6"},"cell_type":"code","source":"# results = tune(\n# Y = df_train.target,\n# X = df_train.question_text,\n# YX = df_train.label_and_text,\n# k = 5, \n# lr = [0.025, 0.05],\n# wordNgrams = [2, 3],\n# epoch = [5, 10, 15, 20],\n# ws=[5, 10, 30],\n# loss=['hs', 'softmax']\n# )","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9bba353f9a6d130e00ffc0f6e2262b992e493b09"},"cell_type":"markdown","source":"Here are the top 10:\n```\n lr wordNgrams epoch loss ws_val mean_acc\n ----------------------------------------------------------------\n0.95515 [0.025, 2, 5, 'softmax', 10, 0.9551512035405768]\n0.95515 [0.025, 3, 5, 'softmax', 10, 0.955158859429776]\n0.95516 [0.01, 2, 10, 'softmax', 30, 0.9551688125617146]\n0.95517 [0.025, 2, 5, 'softmax', 5, 0.9551711096696319]\n0.95521 [0.025, 3, 5, 'softmax', 5, 0.9552124531923372]\n0.95522 [0.025, 3, 10, 'hs', 10, 0.9552247030042806]\n0.95523 [0.01, 2, 15, 'hs', 5, 0.9552384845907967]\n0.95523 [0.05, 2, 5, 'hs', 30, 0.9552392504048104]\n0.95524 [0.05, 3, 5, 'hs', 5, 0.9552400154040317]\n0.95529 [0.025, 3, 10, 'hs', 30, 0.9552905475547778]\n```"},{"metadata":{"_uuid":"386e77f0c7864f92597d8f38d9d2872db2d499f3"},"cell_type":"markdown","source":"# Training\n\nNow that we know what are the best parameters for the FastText algorithm, we are ready to move on to the training section. So far we have referred to the trained model as \"the classifier\", but there are multiple classifiers. Certainly, any classifier could sometimes be wrong and give a wrong predictions, but by increasing the number of classifiers we decrease the chance of predicting wrong, considering the fact that our at least two of our classifiers are good. \n\nWill top 3 different outputs that the `tune` function displayed and use them create classifier.\n```\n lr wordNgrams epoch loss ws_val mean_acc\n ----------------------------------------------------------------\n0.95524 [0.05, 3, 5, 'hs', 5, 0.9552400154040317]\n0.95529 [0.025, 3, 10, 'hs', 30, 0.9552905475547778]\n0.95539 [0.05, 3, 5, 'hs', 5, 0.9553962030503383]\n```"},{"metadata":{"trusted":true,"_uuid":"5c4aca8effcb32f3eae29864e3000e6819c78651"},"cell_type":"code","source":"# train the classifier\nclassifier1 = ft.FastText.train_supervised(\n 'train.txt', lr=0.05, wordNgrams=3, epoch=5, loss='hs', ws=5\n)\nclassifier2 = ft.FastText.train_supervised(\n 'train.txt', lr=0.025, wordNgrams=3, epoch=1, loss='hs', ws=30\n)\nclassifier3 = ft.FastText.train_supervised(\n 'train.txt', lr=0.05, wordNgrams=3, epoch=5, loss='hs', ws=5\n)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"272e3cd5bc644a5db5f83d4c87935403c34dff2f"},"cell_type":"markdown","source":"# Predictions\n\nIntuitively, for every classifier there must be a unique predictions, thus we will create multiple predictions:"},{"metadata":{"trusted":true,"_uuid":"e0127294092f952587bb31d17a5dd8da2ac5133c"},"cell_type":"code","source":"# make predictions for test data\npredictions1 = classifier1.predict(df_test.question_text.tolist())\npredictions2 = classifier2.predict(df_test.question_text.tolist())\npredictions3 = classifier3.predict(df_test.question_text.tolist())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"20997445aaba1308a29c6f8434a0787e31601e7d"},"cell_type":"markdown","source":"Now, that we have our predictions results we have to extract the most common ones. This is done with the methods`Counter` and `.most_common(1)`:"},{"metadata":{"trusted":true,"_uuid":"a171bce2b99f3490dd5a526b837a06f6c5441936"},"cell_type":"code","source":"# Combine predictions\nmost_common = np.array([])\nfor i in range(len(predictions1[0])):\n most_common = np.append(\n most_common, \n Counter([\n predictions1[0][i][0],\n predictions2[0][i][0],\n predictions3[0][i][0]\n ]).most_common(1)[0][0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fb2bfa066931fdbc8de0b535624297e0b5d9ffcd"},"cell_type":"markdown","source":"We are now ready to create a submission file:"},{"metadata":{"trusted":true,"_uuid":"5b09022fdbc4b0cdfdaf57c5d0a4411b7cab493d"},"cell_type":"code","source":"# Write submission file\nsubmit = pd.DataFrame({\n 'qid': df_test.qid,\n 'prediction': pd.Series(most_common)\n})\nsubmit.prediction = submit.prediction.apply(lambda x: re.sub('__label__', '', x))\nsubmit.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74fcd4d154b8f7f63c840cedd5919cd466b33503"},"cell_type":"markdown","source":"At last we can have a quick glance at how the submission file looks like:"},{"metadata":{"trusted":true,"_uuid":"931602e1698a09ac540706992d7826ebb3bf998e","scrolled":true},"cell_type":"code","source":"pd.read_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da81e316e63d5defa2233f6994a10d3d741b1e5b"},"cell_type":"markdown","source":"# Conclusion"},{"metadata":{"_uuid":"cfbdba5e509665a606df26fded9fa80c39fc84a6"},"cell_type":"markdown","source":"There many other methods and algorithms that one can use to solve a classification problem like this one, but we intentionally wanted to use FastText. The reason was that we wanted to observe how well does FastText perform on a real world problem and on a serious competition. Certainly, there are some parts of this notebook that could be improved to get a higher score. For instance the preprocessing could be enhanced, by converting all the numbers to their textual representation, converting plural into singular and etc."},{"metadata":{"_uuid":"276c8a669f95630bf144c2e351d4ad7381d33f4c"},"cell_type":"markdown","source":"# References"},{"metadata":{"_uuid":"fe0f3cdda0c60937c94d0cdc88a917d927681fb9"},"cell_type":"markdown","source":"1. Randi H Griffin. \"Facebook's fastText algorithm\", . Last visited: 2 Feb 2019. \n\n2. KDnuggets. \"Text Data Preprocessing: A Walkthrough in Python\", . Last visited: 2 Feb 2019. "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}