{"cells":[{"metadata":{"_uuid":"c89d216f183696c3414ef1f1265f42b30bd16148"},"cell_type":"markdown","source":"This is an approach to the Quora Insincere Questions Classification competition on Kaggle using the features based on the distribution of scores from a word counting approach in my MCS Thesis [Compact Features for Sentiment Analysis'](http://lgaud.github.io/Papers/Thesis_CompactFeaturesForSentimentAnalysis.pdf), also published in a  [Paper from Canadian AI 2011)](http://lgaud.github.io/Papers/CompactFeaturesForSentimentAnalysis.pdf)\n\nMy original implementation was in Java (using Weka and other APIs), and I hadn't previously implemented in Python. I thought it would be worthwhile trying it out on this dataset, as it's a similar binary problem as the problems I looked at in my thesis, and the approach worked relatively well on noisy texts. I don't expect it to perform amazingly on it's own but it may be useful as part of an ensemble.\n\nI've just implemented here using the parameters I found worked relatively well across the problems I looked at in my thesis and haven't made an effort to tune it.\nParams\nIn my thesis for sentiment (and also subjectivity, agreement, and pleasantness), I focused on using 25 bins with Naive Bayes and SVM classifiers, and I used precision to score (count in positive docs / total count)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.svm import SVC\nfrom sklearn.metrics import accuracy_score, f1_score\nfrom sklearn.pipeline import Pipeline\n\noriginal_train = pd.read_csv('../input/train.csv')\n\nval = original_train.sample(frac=0.2, random_state=1234)\ntrain = original_train.drop(val.index)\nsample = train.sample(frac=0.1, random_state=42) # Small sample of the training set for testing code more quickly\nprint(len(val), len(train), len(sample))\ndel original_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9dc2d9707ea40fff265aa05b43ef197e7832bbb"},"cell_type":"code","source":"sample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6488e5e1d02c9ea398bbee431b204718c2a41543"},"cell_type":"code","source":"# Implement as a subclass of scikit-learn's count vectorizer\nclass HistogramVectorizer(CountVectorizer):\n    def __init__(self, input='content', encoding='utf-8',\n                 decode_error='strict', strip_accents=None, lowercase=True,\n                 preprocessor=None, tokenizer=None, analyzer='word',\n                 stop_words=None, token_pattern=r\"(?u)\\b\\w\\w+\\b\",\n                 ngram_range=(1, 1), max_df=1.0, min_df=1,\n                 max_features=None, vocabulary=None, binary=False,\n                 dtype=np.float64, positive_class_label=1, bins=25):\n        \"\"\"See CountVectorizer for params\n        positive_class_label: The label that specifies the positive class, used in fit\n        bins: the number of features to generate \"\"\" \n        super(HistogramVectorizer, self).__init__(input=input, encoding=encoding, decode_error=decode_error,\n            strip_accents=strip_accents, lowercase=lowercase,\n            preprocessor=preprocessor, tokenizer=tokenizer, analyzer=analyzer,\n            stop_words=stop_words, token_pattern=token_pattern,\n            ngram_range=ngram_range, max_df=max_df, min_df=min_df,\n            max_features=max_features, vocabulary=vocabulary, binary=binary,\n            dtype=dtype)\n        self.positive_class_label = positive_class_label\n        self.bins = bins\n        \n    def fit(self, raw_documents, y):\n        X = super(HistogramVectorizer, self).fit_transform(raw_documents)\n        \n        total_counts = np.sum(X, axis=0)\n        pos = super(HistogramVectorizer, self).transform(raw_documents[y == self.positive_class_label])\n        positive_counts = np.sum(pos, axis=0)\n        self.precision_scores = positive_counts / total_counts\n\n        return self\n        \n    def transform(self, raw_documents, copy=True):\n        X = super(HistogramVectorizer, self).transform(raw_documents)\n        docs, words = X.shape\n        ranges = np.linspace(0, 1, num=self.bins)\n        score_counts = np.zeros((docs, self.bins))\n\n        # Todo look at vectorizing\n        for doc_index in range(0, docs):\n            indices = X[doc_index,:].nonzero()\n            for word_index in indices[1]:    \n                score = self.precision_scores[0, word_index]\n                bin_index = np.searchsorted(ranges, score)\n                score_counts[doc_index, bin_index] += 1\n        \n        normalized = score_counts / np.sum(score_counts, axis=1)[:,None]\n        return np.nan_to_num(normalized)\n            ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"448f49723b6b6b02631de1896452c113ca642c78"},"cell_type":"code","source":"vectorizer = HistogramVectorizer(min_df=2, max_df=0.5)\nclassifier = GaussianNB()\ntrain_df = train\nvectorizer.fit(train_df[\"question_text\"], train_df[\"target\"])\ntrain_features = vectorizer.transform(train_df[\"question_text\"])\nclassifier.fit(train_features, train_df[\"target\"])\ntrain_preds = classifier.predict(train_features)\n\n# Training set performance\nprint(accuracy_score(train_preds, train_df[\"target\"]))\nprint(f1_score(train_preds, train_df[\"target\"]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e79b2f61c3b0ca172965ae1c9f98611cf5c42028"},"cell_type":"code","source":"val_features = vectorizer.transform(val[\"question_text\"])\nval_preds = classifier.predict(val_features)\n\nprint(accuracy_score(val_preds, val[\"target\"]))\nprint(f1_score(val_preds, val[\"target\"]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ca84af717aec9cb884b12b87f10fd299b1bb619"},"cell_type":"markdown","source":"So on it's own this isn't performing terribly well, but maybe with more tweaking it might be useful as part of an ensemble."},{"metadata":{"_uuid":"a99a8969b4cba9907d717ae32b8996f855c19d7b"},"cell_type":"markdown","source":"# Visualizing how documents are represented"},{"metadata":{"trusted":true,"_uuid":"29249606d78dadf5f2424c410c158b17ef3e64d4"},"cell_type":"code","source":"pos = vectorizer.transform(val[val[\"target\"] == 1][\"question_text\"])\nneg = vectorizer.transform(val[val[\"target\"] == 0][\"question_text\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5cabc74363896b6066890e22cdf7514f13108c3c"},"cell_type":"code","source":"import matplotlib\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nmean_pos = pos.mean(axis=0)\nmean_neg = neg.mean(axis=0)\nbar_width = 1/30\nbins = np.linspace(0, 1, 25)\n\ny_dims = (0, 0.4)\nx_dims = (0, 1)\n\nfig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10,3))\n\naxes[0].set_title('Insincere Questions (mean)')\naxes[0].bar(bins, mean_pos, width=bar_width, align='edge')\naxes[0].set_ylim(y_dims)\naxes[0].set_xlim(x_dims)\naxes[1].set_title('Sincere Questions (mean)')\naxes[1].bar(bins, mean_neg, width=bar_width, align='edge')\naxes[1].set_ylim(y_dims)\naxes[1].set_xlim(x_dims)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5030083878f27bb7516b9d88b9c5ee005e31c939"},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10,6))\ny_dims = (0, 0.5)\naxes[0, 0].set_title('Insincere Questions')\naxes[0, 0].bar(bins, pos[0], width=bar_width, align='edge')\naxes[0, 0].set_ylim(y_dims)\naxes[0, 0].set_xlim(x_dims)\naxes[1, 0].bar(bins, pos[1], width=bar_width, align='edge')\naxes[1, 0].set_ylim(y_dims)\naxes[1, 0].set_xlim(x_dims)\naxes[0, 1].set_title('Sincere Questions')\naxes[0, 1].bar(bins, neg[0], width=bar_width, align='edge')\naxes[0, 1].set_ylim(y_dims)\naxes[0, 1].set_xlim(x_dims)\naxes[1, 1].bar(bins, neg[1], width=bar_width, align='edge')\naxes[1, 1].set_ylim(y_dims)\naxes[1, 1].set_xlim(x_dims)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f089fefc010f5c8c252d9ab8f1ea41bbcbcf504"},"cell_type":"markdown","source":"The insincere questions have a distribution that's shifted a little to the right. I notice that the full distribution isn't really being used, which I believe is due to this dataset being quite imbalanced and the scoring method I used here (precision) not really accounting for that."},{"metadata":{"_uuid":"aa81a1dd7d9d8dc9974405a2e7ecb5b0680c7599"},"cell_type":"markdown","source":"\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}