{"cells":[{"metadata":{"_uuid":"e293b2a6439b76e63c43f912b4318324abe121fe"},"cell_type":"markdown","source":"### Import modules"},{"metadata":{"trusted":true,"_uuid":"ffb49befc60867b4e2c6edc46ccec50d0497f39a"},"cell_type":"code","source":"import nltk\nimport pandas as pd\nimport re\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nimport string\n\nstopwords = nltk.corpus.stopwords.words('english')\n\nprint('Imports Complete')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6a10491126171c8d5e74fbc13217555dbcc35a4"},"cell_type":"markdown","source":"### Read Data"},{"metadata":{"trusted":true,"_uuid":"56c9f7289599e7cc15469dbcdbf863ec88c573b2"},"cell_type":"code","source":"#Here is where I read my data into pandas dataframes\ntrain_df = pd.read_csv(\"../input/train.csv\")\ntest_df = pd.read_csv(\"../input/test.csv\")\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"700253a37ad6f83b44bbfdd6b7dde7724f6590d3"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f811c95e1ee1d049192aa91c38eddcb460aa716"},"cell_type":"markdown","source":"### Rebalance and reduce sample\nSo I need to re-balanace the data since there are not an even amount of pos and neg example for training and predictions. Some friends of mine told me to always have a balance in order to prevent bias. I am also limiting the amount of features that are processed. This is due to memery limitations. When you run the code below you will see the total amount of features after the rebalanace is over 80,000. This willnot vectoize using our chosen vectorizer (TF-IDF). I have found that 50,000 is about the limit."},{"metadata":{"trusted":true,"_uuid":"cd3b5f861e9c6c32a01391f944e7a3c132f20f82"},"cell_type":"code","source":"\nfrom matplotlib import pyplot\nimport numpy as np\n%matplotlib inline\n\ncount_target_0, count_target_1 = train_df['target'].value_counts()\n\ntrain_df_target_0 = train_df[train_df['target'] == 0]\ntrain_df_target_1 = train_df[train_df['target'] == 1]\n\ntrain_df_target_0_under = train_df_target_0.sample(count_target_1)\ntrain_df_under = pd.concat([train_df_target_0_under, train_df_target_1], axis=0)\n\ntrain_df_under['target'].value_counts().plot(kind='bar', title='Count (target)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"74438bcc8cbe4a182b4e66d66b78581150cc3534"},"cell_type":"code","source":"train_df_under['target'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d0b52c59a0c2d63eb2c0534480ab70d7693c9909"},"cell_type":"code","source":"sam_train_under = train_df_under.sample(50000)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d056a49129426a13a68fbbe52ecebe252ee2fa03"},"cell_type":"markdown","source":"### Let's get some test/train data\nHere is where I split thattraing 80/20 (as denoted by the .2 in the test_size parameter) into a test set and a training set. This 'hold out' method is common."},{"metadata":{"trusted":true,"_uuid":"17acc9861c1fc798cf0e77f119a444d0ae5154db"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(sam_train_under[['question_text']], sam_train_under['target'], test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cdca5d6f89d591096c370fb116018096dffa0598"},"cell_type":"markdown","source":"### Vectorize\nHere is where I vectorize the words in the questions. I am using teh TF-IDF vectorizer. You could also use an N-gram or CountVectorizer as an alternative. Notice that I am removing the stopwords"},{"metadata":{"trusted":true,"_uuid":"9db6b8ea01706bc2972e8124cf283401da2a67e4"},"cell_type":"code","source":"tfidf_vect = TfidfVectorizer(stop_words='english')\ntfidf_vect_fit = tfidf_vect.fit(X_train['question_text'])\n\ntfidf_train = tfidf_vect_fit.transform(X_train['question_text'])\ntfidf_test = tfidf_vect_fit.transform(X_test['question_text'])\n\nX_train_vect = pd.DataFrame(tfidf_train.toarray())\nX_test_vect =  pd.DataFrame(tfidf_test.toarray())\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ed8313469d49ec7545d637705e2feda58b79c99f"},"cell_type":"markdown","source":"### Model eval\nThis is where the rubber meets the road. I am trying out several different machine learning algorithm's to find which one works the best. Google \"no free lunch machine learning\" and you will understand"},{"metadata":{"trusted":true,"_uuid":"068f060b7c2c0e4a20f84fa6d4b42926954d99ef"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\nfrom sklearn.linear_model import LogisticRegression, SGDClassifier\nfrom sklearn.metrics import precision_recall_fscore_support as score\nimport time","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3bd9a2ca05f82f1ae43ee8a428a8c56fa173c6af"},"cell_type":"markdown","source":"## Start with RandomForest\nrf = RandomForestClassifier(n_estimators=150, max_depth=None, n_jobs=-1)\n\nstart = time.time()\nrf_model = rf.fit(X_train_vect, y_train)\nend = time.time()\nfit_time = (end - start)\n\nstart = time.time()\ny_pred = rf_model.predict(X_test_vect)\nend = time.time()\npred_time = (end - start)\n\nprecision, recall, fscore, train_support = score(y_test, y_pred, pos_label=1, average='binary')\nprint('Fit time: {} / Predict time: {} ---- Precision: {} / Recall: {} / Accuracy: {}'.format(\n    round(fit_time, 3), round(pred_time, 3), round(precision, 3), round(recall, 3), round((y_pred==y_test).sum()/len(y_pred), 3)))"},{"metadata":{"_uuid":"4cf8ab4ed23e7d0ffcfab272871a86773e48c416"},"cell_type":"markdown","source":"## Try stochastic gradient descent "},{"metadata":{"trusted":true,"_uuid":"ad4a1bde5b966953602440c61a38c076f31b9181"},"cell_type":"code","source":"sgd = SGDClassifier()\n\nstart = time.time()\nsgd_model = sgd.fit(X_train_vect, y_train)\nend = time.time()\nfit_time = (end - start)\n\nstart = time.time()\ny_pred = sgd_model.predict(X_test_vect)\nend = time.time()\npred_time = (end - start)\n\nprecision, recall, fscore, train_support = score(y_test, y_pred, pos_label=1, average='binary')\nprint('Fit time: {} / Predict time: {} ---- Precision: {} / Recall: {} / Accuracy: {}'.format(\n    round(fit_time, 3), round(pred_time, 3), round(precision, 3), round(recall, 3), round((y_pred==y_test).sum()/len(y_pred), 3)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4c25a9a7b9b2db1e41ae4617526e53c26de4314"},"cell_type":"code","source":"### Try Logistic regression","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9f844af208cb72de8583d83a1f6235b76ccb5e63"},"cell_type":"code","source":"lr = LogisticRegression(C=0.1, solver='sag')\n\nstart = time.time()\nlr_model = lr.fit(X_train_vect, y_train)\nend = time.time()\nfit_time = (end - start)\n\nstart = time.time()\ny_pred = lr_model.predict(X_test_vect)\nend = time.time()\npred_time = (end - start)\n\nprecision, recall, fscore, train_support = score(y_test, y_pred, pos_label=1, average='binary')\nprint('Fit time: {} / Predict time: {} ---- Precision: {} / Recall: {} / Accuracy: {}'.format(\n    round(fit_time, 3), round(pred_time, 3), round(precision, 3), round(recall, 3), round((y_pred==y_test).sum()/len(y_pred), 3)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3fe22ad8ac4c116d072ad42a3c67bfd9ae3fe5d8"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"776afa9ac731fca267f238010adefa9f152f7f0c"},"cell_type":"markdown","source":"## Try GradientBoost\ngb = GradientBoostingClassifier(n_estimators=150, max_depth=11)\n\nstart = time.time()\ngb_model = gb.fit(X_train_vect, y_train)\nend = time.time()\nfit_time = (end - start)\n\nstart = time.time()\ny_pred = gb_model.predict(X_test_vect)\nend = time.time()\npred_time = (end - start)\n\nprecision, recall, fscore, train_support = score(y_test, y_pred, pos_label=1, average='binary')\nprint('Fit time: {} / Predict time: {} ---- Precision: {} / Recall: {} / Accuracy: {}'.format(\n    round(fit_time, 3), round(pred_time, 3), round(precision, 3), round(recall, 3), round((y_pred==y_test).sum()/len(y_pred), 3)))"},{"metadata":{"trusted":true,"_uuid":"a295d92deec84b5ca919f856c33ddcb03f4f50e0"},"cell_type":"markdown","source":"## Try XGBoost\nimport xgboost as xgb\ngbm = xgb.XGBClassifier(n_job=-1)\n\nstart = time.time()\ngbm_model = gbm.fit(X_train_vect, y_train)\nend = time.time()\nfit_time = (end - start)\n\nstart = time.time()\ny_pred = gbm_model.predict(X_test_vect)\nend = time.time()\npred_time = (end - start)\n\nprecision, recall, fscore, train_support = score(y_test, y_pred, pos_label=1, average='binary')\nprint('Fit time: {} / Predict time: {} ---- Precision: {} / Recall: {} / Accuracy: {}'.format(\n    round(fit_time, 3), round(pred_time, 3), round(precision, 3), round(recall, 3), round((y_pred==y_test).sum()/len(y_pred), 3)))"},{"metadata":{"_uuid":"b40ca94f4cd3f6395043a1ed64e38654ee26d0e0"},"cell_type":"markdown","source":"**I have turned the code blocks for the RandomForest, GradientBoost and XGBoost into markdown. I did this since RF and GB takes over an hour to process and XGB runs our of memory. LogisticRegression runs REALLY fast if you use the default solver (lbfgs) and a bit slower in my case when I chose 'sag' but also had the best results**"},{"metadata":{"_uuid":"b98e1b31be4183c378b2513b6da2ae4ce0b7a9b2"},"cell_type":"markdown","source":"### Build Submission \nThis will load the test data and predict the values using our chosen classifier. Once that is complete, I create a submission CSV to evaluatemy results."},{"metadata":{"trusted":true,"_uuid":"12ae45512f7231fb95bc302d4113d31cd58a8eb6"},"cell_type":"code","source":"X_submission = tfidf_vect.transform(test_df['question_text'])\npredicted_test = sgd.predict(X_submission)\n\ntest_df['prediction'] = predicted_test\nsubmission = test_df.drop(columns=['question_text'])\nsubmission.head()\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c50b22b35108c57a25124cd0aace4a39173c0501"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}