{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Read In Data"},{"metadata":{"trusted":true,"_uuid":"e70c0b2e58461846fa9bbffced71911dd426c318"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ndf = train.copy()\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"188bd2adcb92df8099eec5e741bc5941119b744a"},"cell_type":"code","source":"sample = pd.read_csv('../input/sample_submission.csv')\nsample.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f48286979dff2ea18360712fd48bc4bf19228922"},"cell_type":"markdown","source":"# Understanding the Data"},{"metadata":{"trusted":true,"_uuid":"c9029ebd191bc11bcb779502148480fe3746c8c8"},"cell_type":"code","source":"train.groupby('target').count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"033ea09634d6f274fa5789778a1ccfd56c84180d"},"cell_type":"code","source":"train.groupby('target').count() / train.shape[0] * 100","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e290b463137b1318db23be9e0ac6f2c85f4ee074"},"cell_type":"markdown","source":"This is a very imbalanced dataset, with only 6% of the records labelled as insincere."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"8d47a29835093c893c8a46c6703acf6676587ec5"},"cell_type":"code","source":"list(train.loc[train.target == 1].question_text.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"632c569c7d17a564fae5666f4316e5e039a0d5f1"},"cell_type":"code","source":"list(train.loc[train.target == 1].question_text.sample(10,random_state=504))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"85f20d25f4c8852e64b936b9779af541103e3fe1"},"cell_type":"markdown","source":"It looks to me like \"insincere\" is, from Quora's perspective, questions highly lacking in quality. Some of these look like something a kid would post to try to be funny, but others look like they could be sincere musings of the misinformed. For example, religious riots in India are a very complicated topic, and to assume that a question about any given religious sect being involved in one of them is a conspiracy theory question rather than a question about the complexities of Indian pluralism is not the best idea in my opinion. Also, in Buddhism there actually are questions around whether a female can attain Nirvana. Sexist? Yes. Question rooted in real people's real beliefs over the last few thousand years? Also yes. \"Sincerity\" for Quora's purposes is quite different from \"sincerity\" for, say, Reddit. This should be kept in mind in our feature selection process."},{"metadata":{"trusted":true,"_uuid":"b90f597b02c21d33bf4fd32d9fb69b8f8f7882cc"},"cell_type":"code","source":"list(train.loc[train.target == 0].question_text.sample(15,random_state=405))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d21c2fda1917c786246b7f8a08b34f2f699c986d"},"cell_type":"markdown","source":"\"My desk turned into a Viking ship after listening to Amon Amarth. What should I do with it?\" is obviously an insincere question; I don't understand these labels at all. I think this data is poorly labelled."},{"metadata":{"_uuid":"40a4b91f166a9b67ea677a4d0358872a83f7450c"},"cell_type":"markdown","source":"# What makes a question \"insincere\"? Hypotheses...\n\n_In general, not always_\n\n* Simpler vocabulary\n    * capture variety of words?\n* Sweeping generalizations\n    * occurence of an ethnic group and a political group in one question, for example. Not sure how to capture this.\n* Poor grammar\n    * bigrams and/or trigrams can capture this?\n* Profanity and inflammatory words\n    * collisions with urban dictionary or something like that?\n* Emotional words?\n    * no evidence for this yet, just a thought"},{"metadata":{"_uuid":"5c4fb3254129a1eb332f9053b768dfe9c949e95b"},"cell_type":"markdown","source":"# Text Feature Extraction"},{"metadata":{"trusted":true,"_uuid":"c3b279abb6295f7411e9bdbf094a92685fd64328"},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer\nfrom stop_words import get_stop_words\nimport re\n\nstops = get_stop_words('english')\ndef clean_list(x):\n    x = [i.strip() for i in x if i.strip() not in stops and i.strip() != '']\n    return x\n    \ntrain.question_text = train.question_text.apply(lambda x: x.lower().strip())\ntrain.question_text = train.question_text.apply(lambda x: re.sub(r'[?,\\.!\\\"\\']',' ', x))\ntrain.question_text = train.question_text.apply(lambda x: x.split(' '))\ntrain.question_text = train.question_text.apply(clean_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"610390d35c3b1e2e6d13df4d2666ba428d83c4be"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"a41032d3b8ea8aa16f6a4a0c485ed7f53fd76936"},"cell_type":"code","source":"from nltk.stem import WordNetLemmatizer\n\nlemmatizer = WordNetLemmatizer()\ntrain.question_text = train.question_text.apply(lambda x: [lemmatizer.lemmatize(i) for i in x])\ntrain['preprocessed_text'] = train.question_text.apply(lambda x: ' '.join(x))\ntrain.question_text = df.question_text\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4ceb590be8015c4c3fb0c4682aca926585b440b"},"cell_type":"code","source":"import spacy\nfrom spacy import displacy\nfrom collections import Counter\nimport en_core_web_sm\nnlp = en_core_web_sm.load()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac8d8d28d284aa0f62aa2801b3900663cd00497f"},"cell_type":"code","source":"train_pos = train.loc[train.target == 1].sample(5000,random_state=5094)\ntrain_neg = train.loc[train.target == 0].sample(5000,random_state=5094)\ntest_pos = train.loc[(train.target == 1) & ~train.qid.isin(train_pos.qid)].sample(5000,random_state=5094)\ntest_neg = train.loc[(train.target == 0) & ~train.qid.isin(train_neg.qid)].sample(5000,random_state=5094)\ntrain_subset = pd.concat([train_pos,train_neg,test_pos,test_neg])\ntrain_subset['ner_data'] = train_subset.question_text.apply(lambda x: nlp(x).ents)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"83dc9f6a837e80e90d5d0eb1cbb2bbdcda18ee24"},"cell_type":"code","source":"def get_ne_counts(tuples):\n    nes = {}\n    for tup in tuples:\n        info = (tup.text, tup.label_)\n        if info[1] in nes:\n            nes[info[1]] += 1\n        else:\n            nes[info[1]] = 1\n    return nes\n\ntrain_subset['ne_types'] = train_subset.ner_data.apply(get_ne_counts)\nne_df = pd.DataFrame(list(train_subset['ne_types']),index=train_subset.index)\nne_df = ne_df.fillna(0)\ntrain_subset = pd.concat([train_subset,ne_df],axis=1,join_axes=[train_subset.index])\ntrain_subset.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"28aa4ac77f3bb1e512848e94adad57361d46a3b2"},"cell_type":"code","source":"cvec = CountVectorizer()\ncounts = cvec.fit_transform(train_subset.preprocessed_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"49d5865b59aee3c57dd65fb46247e06e72dc986c"},"cell_type":"code","source":"counts = pd.DataFrame(counts.todense(),index=train_subset.index)\ncounts.columns = cvec.get_feature_names()\ncounts = counts[[c for c in counts.columns if counts[c].sum() >= 3]]\ncounts.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"56458169229c9f8d2c1fdbc0eadd5d49ea32bcb8"},"cell_type":"code","source":"counts.to_csv('counts.csv',index=True,header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"00d2598f1aaa71e0e7c1f39231e2f097fcfeb469"},"cell_type":"code","source":"feature_df = pd.concat([train_subset.iloc[:,6:],counts],axis=1)\nfeature_df['target'] = list(train_subset['target'])\nfeature_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4543e88c99c5f1e75f171502e4222a6e9ef96b8d"},"cell_type":"markdown","source":"# Train Model"},{"metadata":{"trusted":true,"_uuid":"76686272b46f53b1c13415b2bbb2e0333ef38fd8"},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nrfc = RandomForestClassifier(random_state=40938)\n\nX = feature_df[[c for c in feature_df.columns if c != 'target']]\ny = feature_df.target\nX_train = X.iloc[:10000,:]\nX_test = X.iloc[10000:,:]\ny_train = y.iloc[:10000]\ny_test = y.iloc[10000:]\n\nrfc.fit(X_train,y_train)\npredictions = rfc.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"34a5330b7415bcf9d9c596b90cbac255dda30e11"},"cell_type":"code","source":"eval_df = pd.DataFrame({'actual': list(y_test),\n                       'predicted': predictions})\neval_df['incorrect'] = eval_df['actual'] - eval_df['predicted']\neval_df.incorrect = eval_df.incorrect.apply(abs)\n1 - eval_df.incorrect.sum() / eval_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b4612bc25a3861fc0b62dd9e41861657cdf6eab"},"cell_type":"code","source":"from sklearn.model_selection import RandomizedSearchCV\nfrom pprint import pprint\n\n# Number of trees in random forest\nn_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 100)]\n# Number of features to consider at every split\nmax_features = ['auto', 'sqrt']\n# Maximum number of levels in tree\nmax_depth = [int(x) for x in np.linspace(10, 100, num = 10)]\nmax_depth.append(None)\n# Minimum number of samples required to split a node\nmin_samples_split = [2, 5, 10]\n# Minimum number of samples required at each leaf node\nmin_samples_leaf = [1, 2, 4]\n# Method of selecting samples for training each tree\nbootstrap = [True, False]\n# Create the random grid\nrandom_grid = {'n_estimators': n_estimators,\n               'max_features': max_features,\n               'max_depth': max_depth,\n               'min_samples_split': min_samples_split,\n               'min_samples_leaf': min_samples_leaf,\n               'bootstrap': bootstrap}\npprint(random_grid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d932983780979c64a6ed75e6d1176ea3e31ea68","scrolled":false},"cell_type":"code","source":"# Use the random grid to search for best hyperparameters\n# First create the base model to tune\n# Random search of parameters, using 3 fold cross validation, \n# search across 100 different combinations, and use all available cores\nrf_random = RandomizedSearchCV(estimator = rfc, n_iter=1,param_distributions = random_grid, \n                               cv = 3, verbose=2, random_state=42, n_jobs = -1)\n# Fit the random search model\nrf_random.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"43ee144debf35bcd18c6e1546550186bf184a8e2"},"cell_type":"code","source":"rf_random.best_params_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"72df0562badeccd5c0e8224d160bad92301b8165"},"cell_type":"code","source":"def evaluate(model, test_features, test_labels):\n    predictions = model.predict(test_features)\n    errors = sum(abs(predictions - test_labels)) / len(test_labels)\n    accuracy = 100 - errors\n    print('Model Performance')\n    print('Accuracy = {:0.2f}%.'.format(accuracy))\n    \n    return accuracy\nbase_model = RandomForestClassifier(n_estimators = 10, random_state = 42)\nbase_model.fit(X_train, y_train)\nbase_accuracy = evaluate(base_model, X_test, y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78fa9205937905d0bdc98e20d240e821cf3f4663"},"cell_type":"code","source":"best_random = rf_random.best_estimator_\nrandom_accuracy = evaluate(best_random, X_test, y_test)\nprint('Improvement of {:0.2f}%.'.format( 100 * (random_accuracy - base_accuracy) / base_accuracy))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1788334b78939d7e8d459114771721352c11e314"},"cell_type":"code","source":"predictions = best_random.predict(X_test)\neval_df = pd.DataFrame({'actual': list(y_test),\n                       'predicted': predictions})\neval_df['incorrect'] = eval_df['actual'] - eval_df['predicted']\neval_df.incorrect = eval_df.incorrect.apply(abs)\n1 - eval_df.incorrect.sum() / eval_df.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"737da2e6822a42b55eeaf106354762de8e70aa6f"},"cell_type":"code","source":"precision = eval_df.loc[(eval_df.actual == 1) & (eval_df.predicted == 1)].shape[0] / \\\neval_df.loc[(eval_df.predicted == 1)].shape[0]\nprecision","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d19dc0ebd523dafccab0d1afbdc467f28984e20"},"cell_type":"code","source":"recall = eval_df.loc[(eval_df.actual == 1) & (eval_df.predicted == 1)].shape[0] / \\\neval_df.loc[(eval_df.actual == 1)].shape[0]\nrecall","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2ffcdf93d97a4c6273df463ce64cb817e63eb3b5"},"cell_type":"code","source":"test_data = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de480ab99b4b2cb5bd6b253e62cc89af03c5bcc0"},"cell_type":"markdown","source":"# Make Predictions"},{"metadata":{"trusted":true,"_uuid":"f8c336f9fda33af723b464049288ea32b74b67bc"},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"779aab99cd8cc493353bdcf554f4614bbe6af28d"},"cell_type":"code","source":"original = list(test_data['question_text'])\ntest_data.question_text = test_data.question_text.apply(lambda x: x.lower().strip())\ntest_data.question_text = test_data.question_text.apply(lambda x: re.sub(r'[?,\\.!\\\"\\']',' ', x))\ntest_data.question_text = test_data.question_text.apply(lambda x: x.split(' '))\ntest_data.question_text = test_data.question_text.apply(clean_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"a146d174f1a534cc52e65d9b9870c28697f1f045"},"cell_type":"code","source":"test_data.question_text = test_data.question_text.apply(lambda x: [lemmatizer.lemmatize(i) for i in x])\ntest_data['preprocessed_text'] = test_data.question_text.apply(lambda x: ' '.join(x))\ntest_data.question_text = original\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6a1302335af5acb79ef9c2a11736c81241b3ed5c"},"cell_type":"code","source":"test_data['ner_data'] = test_data.question_text.apply(lambda x: nlp(x).ents)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dab64142bcb1267b775ad2cda6207b0da33d1efe"},"cell_type":"code","source":"test_data['ne_types'] = test_data.ner_data.apply(get_ne_counts)\nne_df = pd.DataFrame(list(test_data['ne_types']),index=test_data.index)\nne_df = ne_df.fillna(0)\ntest_data = pd.concat([test_data,ne_df],axis=1,join_axes=[test_data.index])\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5d98145d6b75e42f423633fbbbe8e3f9bbf9c6c"},"cell_type":"code","source":"counts = cvec.transform(test_data.preprocessed_text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"a1da6849ee2042caac7536389bb3bbd4846db2b9"},"cell_type":"code","source":"import gc\n\npredictions = []\nfor i in range(5000,counts.shape[0]+4999,5000):\n    gc.collect()\n    subset = counts[i-5000:i].todense()\n    subset = pd.DataFrame(subset,index=test_data.index[i-5000:i])\n    subset.columns = cvec.get_feature_names()\n    subset = subset[[c for c in subset.columns if c in X_train.columns]]\n    feature_df = pd.concat([test_data.iloc[i-5000:i,6:],subset],axis=1)\n    for col in X_train.columns:\n        if col not in feature_df.columns:\n            feature_df[col] = 0\n    predictions += list(best_random.predict(feature_df))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4087561a37a8a05d593ab355c07f134ce52b790"},"cell_type":"code","source":"test_data = pd.read_csv('../input/test.csv')\ntest_data['prediction'] = predictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0184c02ba74bf31104981c4ba16cfd7a225577df"},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48b0d89b582ef8fb1d4e7c8d7ee3a93ff2f40879"},"cell_type":"code","source":"test_data.groupby('prediction').count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1c76402920df5ca0ff0186da86d0f91bc37ed314"},"cell_type":"code","source":"test_data = test_data[['qid','question_text']]\ntest_data.to_csv('submission.csv',index=None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ed69d021bf927268fc33c6b91f1f126b2c78be8"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}