{"cells":[{"metadata":{"_uuid":"2545a3ad7f22c0d72d465b639be421320a7d0896"},"cell_type":"markdown","source":"# Quora Insincere Questions Logistic Regression with Upsampling\n\nIn our previous kernel we learned that the models were not generalizing well to the the competition's test set.  We will try to overcome this by upsampling the minority class.  Other major changes from the [downsampling kernel](https://www.kaggle.com/tboyle10/sklearn-baseline-models) include eliminating the use of spacey and adding ngrams.  Even with downsampling in the previous kernel, text preprocessing and lemmatization with spacey took a very long time.  We'll try using gensim's preprocess_string which uses stemming to decrease our text processing time.  After cleaning and stemming the text we will also create ngrams to hopefully improve our model performance."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# import packages\nimport numpy as np\nimport pandas as pd\nimport re\n\nfrom gensim import corpora, models, similarities\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import f1_score\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\n\nnp.random.seed(27)\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eaddf3f534ef2e9fe14142e844ce12426453961b"},"cell_type":"markdown","source":"#### Text Pre-processing"},{"metadata":{"trusted":true,"_uuid":"ee99bc21de267796eccacc8941a35d4e1667f6de"},"cell_type":"code","source":"contractions = {\n\"ain't\": \"is not\",\n\"aren't\": \"are not\",\n\"can't\": \"cannot\",\n\"can't've\": \"cannot have\",\n\"'cause\": \"because\",\n\"could've\": \"could have\",\n\"couldn't\": \"could not\",\n\"couldn't've\": \"could not have\",\n\"didn't\": \"did not\",\n\"doesn't\": \"does not\",\n\"don't\": \"do not\",\n\"hadn't\": \"had not\",\n\"hadn't've\": \"had not have\",\n\"hasn't\": \"has not\",\n\"haven't\": \"have not\",\n\"he'd\": \"he would\",\n\"he'd've\": \"he would have\",\n\"he'll\": \"he will\",\n\"he'll've\": \"he he will have\",\n\"he's\": \"he is\",\n\"how'd\": \"how did\",\n\"how'd'y\": \"how do you\",\n\"how'll\": \"how will\",\n\"how's\": \"how is\",\n\"I'd\": \"I would\",\n\"I'd've\": \"I would have\",\n\"I'll\": \"I will\",\n\"I'll've\": \"I will have\",\n\"I'm\": \"I am\",\n\"I've\": \"I have\",\n\"i'd\": \"i would\",\n\"i'd've\": \"i would have\",\n\"i'll\": \"i will\",\n\"i'll've\": \"i will have\",\n\"i'm\": \"i am\",\n\"i've\": \"i have\",\n\"isn't\": \"is not\",\n\"it'd\": \"it would\",\n\"it'd've\": \"it would have\",\n\"it'll\": \"it will\",\n\"it'll've\": \"it will have\",\n\"it's\": \"it is\",\n\"let's\": \"let us\",\n\"ma'am\": \"madam\",\n\"mayn't\": \"may not\",\n\"might've\": \"might have\",\n\"mightn't\": \"might not\",\n\"mightn't've\": \"might not have\",\n\"must've\": \"must have\",\n\"mustn't\": \"must not\",\n\"mustn't've\": \"must not have\",\n\"needn't\": \"need not\",\n\"needn't've\": \"need not have\",\n\"o'clock\": \"of the clock\",\n\"oughtn't\": \"ought not\",\n\"oughtn't've\": \"ought not have\",\n\"shan't\": \"shall not\",\n\"sha'n't\": \"shall not\",\n\"shan't've\": \"shall not have\",\n\"she'd\": \"she would\",\n\"she'd've\": \"she would have\",\n\"she'll\": \"she will\",\n\"she'll've\": \"she will have\",\n\"she's\": \"she is\",\n\"should've\": \"should have\",\n\"shouldn't\": \"should not\",\n\"shouldn't've\": \"should not have\",\n\"so've\": \"so have\",\n\"so's\": \"so as\",\n\"that'd\": \"that would\",\n\"that'd've\": \"that would have\",\n\"that's\": \"that is\",\n\"there'd\": \"there would\",\n\"there'd've\": \"there would have\",\n\"there's\": \"there is\",\n\"they'd\": \"they would\",\n\"they'd've\": \"they would have\",\n\"they'll\": \"they will\",\n\"they'll've\": \"they will have\",\n\"they're\": \"they are\",\n\"they've\": \"they have\",\n\"to've\": \"to have\",\n\"wasn't\": \"was not\",\n\"we'd\": \"we would\",\n\"we'd've\": \"we would have\",\n\"we'll\": \"we will\",\n\"we'll've\": \"we will have\",\n\"we're\": \"we are\",\n\"we've\": \"we have\",\n\"weren't\": \"were not\",\n\"what'll\": \"what will\",\n\"what'll've\": \"what will have\",\n\"what're\": \"what are\",\n\"what's\": \"what is\",\n\"what've\": \"what have\",\n\"when's\": \"when is\",\n\"when've\": \"when have\",\n\"where'd\": \"where did\",\n\"where's\": \"where is\",\n\"where've\": \"where have\",\n\"who'll\": \"who will\",\n\"who'll've\": \"who will have\",\n\"who's\": \"who is\",\n\"who've\": \"who have\",\n\"why's\": \"why is\",\n\"why've\": \"why have\",\n\"will've\": \"will have\",\n\"won't\": \"will not\",\n\"won't've\": \"will not have\",\n\"would've\": \"would have\",\n\"wouldn't\": \"would not\",\n\"wouldn't've\": \"would not have\",\n\"y'all\": \"you all\",\n\"y'all'd\": \"you all would\",\n\"y'all'd've\": \"you all would have\",\n\"y'all're\": \"you all are\",\n\"y'all've\": \"you all have\",\n\"you'd\": \"you would\",\n\"you'd've\": \"you would have\",\n\"you'll\": \"you will\",\n\"you'll've\": \"you will have\",\n\"you're\": \"you are\",\n\"you've\": \"you have\"\n}\n\nc_re = re.compile('(%s)' % '|'.join(contractions.keys()))\n\ndef expandContractions(text, c_re=c_re):\n    def replace(match):\n        return contractions[match.group(0)]\n    return c_re.sub(replace, text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d2dd1693f3eccef038cff273125e934b55af1003"},"cell_type":"code","source":"# function to clean and lemmatize text and remove stopwords\nfrom gensim.parsing.preprocessing import preprocess_string\n\ndef gensim_preprocess(docs):\n    docs = [expandContractions(doc) for doc in docs]\n    docs = [preprocess_string(text) for text in docs]\n    return pd.Series(docs)\n\ngensim_preprocess(train.question_text.iloc[10:15])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cbef750800237534670c371bcf4103d18e06529d"},"cell_type":"code","source":"# apply text-preprocessing function to training set\n%time train_corpus = gensim_preprocess(train.question_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ce2d567ac6c5acd054db57b1668afed85238c9f"},"cell_type":"code","source":"def create_ngrams(docs):\n    # create the bigram and trigram models\n    bigram = models.Phrases(docs, min_count=1, threshold=1)\n    trigram = models.Phrases(bigram[docs], min_count=1, threshold=1)  \n    # phraser is faster\n    bigram_mod = models.phrases.Phraser(bigram)\n    trigram_mod = models.phrases.Phraser(trigram)\n    # apply to docs\n    docs = trigram_mod[bigram_mod[docs]]\n    return docs\n\ntrain_texts = create_ngrams(train_corpus)\ntrain_texts[81]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"583d3666f78e0c4a4c881020462a77fdb75e2a64"},"cell_type":"code","source":"# preparing ngrams for modeling\ntrain_texts = [' '.join(text) for text in train_texts]\ntrain['ngrams'] = train_texts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19d31ce653bdea205065fc0491d0d3676c35fe9e"},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\n\ntv = TfidfVectorizer(use_idf=True,\n                     min_df=50,\n                     max_features=20000,\n                     ngram_range=(1,2),\n                     norm='l1',\n                     smooth_idf=True).fit(train_texts)\ntv_matrix = tv.transform(train_texts)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a3c3d335917aa0318af463c7a5416158a3d5b4f"},"cell_type":"code","source":"# print target counts\ntrain.target.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64b162511331be4ee3ac1418ce6850fff86edc5a"},"cell_type":"code","source":"# Upsampling\nfrom imblearn.over_sampling import RandomOverSampler\nros = RandomOverSampler(sampling_strategy={1:1225312, # upsample minority class to equal majority count\n                                          })\nX, y = ros.fit_sample(tv_matrix, train.target)\n\n# split into train/test sets\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7bc9ba1b315c6b1b212483f6a1d34ffe2f406d7a"},"cell_type":"markdown","source":"## Logistic Regression Model"},{"metadata":{"trusted":true,"_uuid":"e0df2aecb50730e61c034849bdcf2f7e4859e379"},"cell_type":"code","source":"\nfrom sklearn.linear_model import LogisticRegression\nlr = LogisticRegression()\nlr.fit(X_train, y_train)\n\ny_ = lr.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48b67400149772936e26fcfc948c31e34f3ca7c4"},"cell_type":"code","source":"print('Logistic Regression Score: ', f1_score(y_, y_test))\n\nprint(classification_report(y_test, y_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2178c186bf7d20e290985e7a5ee38101506b6926"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nconfusion_matrix(y_test, y_)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8c7f2053388eda0ed2dd6d6a80371ed43301c76"},"cell_type":"markdown","source":"## Bernoulli Naive Bayes Model"},{"metadata":{"trusted":true,"_uuid":"25a1d13d96132b1a9b17a9eb2e5837cf4c6f0d80"},"cell_type":"code","source":"from sklearn.naive_bayes import BernoulliNB\n\nbnb = BernoulliNB()\n%time bnb.fit(X_train, y_train)\ny_ = bnb.predict(X_test)\n\nprint('Naive Bayes Score: ', f1_score(y_, y_test))\n\nprint(classification_report(y_test, bnb_y_))\n\npd.DataFrame(confusion_matrix(y_test, bnb_y_))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14537a2a995adec59e18c8a11ef2b340b31d5b2c"},"cell_type":"markdown","source":"## XGBoost Model"},{"metadata":{"trusted":true,"_uuid":"90ebcdbd9322665e0248b5216ab8537d41cfb27b"},"cell_type":"code","source":"import xgboost as xgb\n\nxgb_model = xgb.XGBClassifier().fit(X_train, y_train)\ny_ = xgb_model.predict(X_test)\n\nprint('XGBoost Score: ', f1_score(y_, y_test))\n\nprint(classification_report(y_test, xgb_y_))\n\npd.DataFrame(confusion_matrix(y_test, xgb_y_))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb20359980ec7fb6c3fa6c543f6f8a57726dd4c9"},"cell_type":"markdown","source":"## Ensemble Model"},{"metadata":{"trusted":true,"_uuid":"3b50d68678cc09c2b3806143bb4c05e97b7dc3c3"},"cell_type":"code","source":"from sklearn.ensemble import VotingClassifier\n\n#create submodels\nestimators = []\n\nmodel1 = lr\nmodel2 = bnb\nmodel3 = xgb_model\n\n\nestimators.append(('logistic', model1))\nestimators.append(('bernoulli', model2))\nestimators.append(('xgboost', model3))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b6d06eeab7f649ced20ec3f11c935d6e42f6515"},"cell_type":"code","source":"# create ensemble model\n%time ensemble = VotingClassifier(estimators).fit(X_train, y_train)\ny_ = ensemble.predict(X_test)\nprint('Ensemble Score: ', f1_score(y_, y_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fad6fb749ffa5e288db76beb568420ee34a9bb60"},"cell_type":"code","source":"%time ensemble_y_ = ensemble.predict(X_test)\nprint(classification_report(y_test, ensemble_y_))\n\npd.DataFrame(confusion_matrix(y_test, ensemble_y_))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3eb85e1f3a30c133c5b751aebec4ffdaf9ab2aec"},"cell_type":"markdown","source":"## Interpret Results\n\nLogistic Regression F1: .760  \nNaive Bayes F1: .766  \nXGBoost F1: .469   \nEnsemble F1: .755  \n\nWe can see here our F1 scores have decreased significantly as compared to the downsampling kernel.  When submitting to the competition this kernel underperformed our previous attempt with a public score of only 0.170\n\n## Submit to Competition"},{"metadata":{"trusted":true,"_uuid":"2b69b74d425095ce70f1677ff575f92766e80e91"},"cell_type":"code","source":"# preprocessing/lemmatizing/stemming test data\n%time test_corpus = gensim_preprocess(test.question_text)\ntest_texts = create_ngrams(test_corpus)\n\ntest_texts = [' '.join(text) for text in test_texts]\ntest['ngrams'] = test_texts\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"52fabf6a77b62f3f53042b1d7a1e09bf8694f54a"},"cell_type":"code","source":"# ensemble on test data\n# fit on whole training set\nensemble.fit(tv_matrix, train.target)\nprediction = ensemble.predict(tv.transform(test.ngrams))\n\nsubmission = pd.DataFrame({'qid':test.qid, 'prediction':prediction})\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}