{"cells":[{"metadata":{"_uuid":"76f904271d0f2cb0c41af53a8f2433a98c4abe3a"},"cell_type":"markdown","source":"**The main purpose of this kernel to show that there are a lot of other ways to vectorize texts, not only tfidf.**\n\nIt is common to use TFIDF in text-type competions on Kaggle, but as far as we are solving classification task we have much more types of text vectorization: \n1. TFICF: tf & inverse category frequency\n2. TFOR: tf & odds ratio\n3. TFRF: tf & relevance frequency\n\nMore detailed examples and implementation you could find at [Textvec](https://github.com/zveryansky/textvec) (commits and stars are welcomed!)\n\nTLDR: you can use Textvec like sklearn TfidfVectorizer and add to blending."},{"metadata":{"_uuid":"114e9957a6ce16dadd2e17b465a704cd020ed9ac"},"cell_type":"markdown","source":"**TFOR (Odds ratio) explanation:**\n\nWhat is odds ratio? Wiki: [The odds ratio (OR) is a statistic defined as the ratio of the odds of A in the presence of B and the odds of A without the presence of B. This statistic attempts to quantify the strength of the association between A and B.](https://en.wikipedia.org/wiki/Odds_ratio)\n\nIn general this mean that for every word we could count the odds of label 1 if this word is in text.\n\nHere is an example of using OR for dimension reduction with CI:"},{"metadata":{"trusted":true,"_uuid":"0ec011b819aa88de0ffc23676742668d6dd9ffb4"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import f1_score, roc_auc_score\nfrom sklearn.model_selection import StratifiedKFold, train_test_split\nfrom textvec import vectorizers\n\n\ntrain = pd.read_csv('../input/train.csv').fillna(' ')#.sample(10000, random_state=13)\ntrain_target = train['target'].values\n\ntrain_text = train['question_text']\n\nX_train, X_test, y_train, y_test = train_test_split(train_text, train_target, test_size=0.1, random_state=13)\n\ncount_vec = CountVectorizer(strip_accents='unicode',\n    token_pattern=r'\\w{1,}',\n    ngram_range=(1, 1)).fit(X_train)\n\ntfor_vec = vectorizers.TforVectorizer(sublinear_tf=True)\ntfor_vec.fit(count_vec.transform(X_train), y_train)\ntrain_or, ci_95 = tfor_vec.transform(count_vec.transform(X_train), confidence=True)\ntest_or = tfor_vec.transform(count_vec.transform(X_test))\n\nclassifier = LogisticRegression(C=10, solver='sag', random_state=13)\nclassifier.fit(train_or, y_train)\nval_preds = classifier.predict_proba(test_or)[:,1]\nprint('ROC_AUC -> ', roc_auc_score(y_test, val_preds))\nprint('shape -> ', train_or.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"00f5d9d3a6e9dc877a4e982159e48f84bb99f006","scrolled":true},"cell_type":"code","source":"classifier = LogisticRegression(C=10, solver='sag', random_state=13)\nclassifier.fit(train_or[:,ci_95], y_train)\nval_preds = classifier.predict_proba(test_or[:,ci_95])[:,1]\nprint('ROC_AUC -> ', roc_auc_score(y_test, val_preds))\nprint('shape -> ', train_or[:,ci_95].shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c10747d8dd4558a6b5214814cf1b600cfa90f05c"},"cell_type":"markdown","source":"As you could see we achieved nearly the same score but with 8 times smaller dimension.\n\nNow lets test the correlation with TFIDF:"},{"metadata":{"trusted":true,"_uuid":"3bf01ae82d83edddc2b03915c9ef1bd1f392582a","scrolled":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import f1_score, roc_auc_score\nfrom sklearn.model_selection import StratifiedKFold\nfrom textvec import vectorizers\n\n\ntrain = pd.read_csv('../input/train.csv').fillna(' ')#.sample(100000, random_state=13)\ntest = pd.read_csv('../input/test.csv').fillna(' ')#.sample(10000, random_state=13)\ntest_qid = test['qid']\ntrain_target = train['target'].values\n\ntrain_text = train['question_text']\ntest_text = test['question_text']\n\ntfidf_vec = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents='unicode',\n    token_pattern=r'\\w{1,}',\n    ngram_range=(1, 1))\ntfidf_vec.fit(pd.concat([train_text, test_text]))\ntrain_idf = tfidf_vec.transform(train_text)\n\n\ncount_vec = CountVectorizer(strip_accents='unicode',\n    token_pattern=r'\\w{1,}',\n    ngram_range=(1, 1)).fit(train_text)\n\ntfrf_vec = vectorizers.TfrfVectorizer(sublinear_tf=True)\ntfrf_vec.fit(count_vec.transform(train_text), train_target)\ntrain_rf = tfrf_vec.transform(count_vec.transform(train_text))\n\ntfor_vec = vectorizers.TforVectorizer(sublinear_tf=True)\ntfor_vec.fit(count_vec.transform(train_text), train_target)\ntrain_or = tfor_vec.transform(count_vec.transform(train_text))\n\ntficf_vec = vectorizers.TfIcfVectorizer(sublinear_tf=True)\ntficf_vec.fit(count_vec.transform(train_text), train_target)\ntrain_icf = tficf_vec.transform(count_vec.transform(train_text))\n\ntfbinicf_vec = vectorizers.TfBinIcfVectorizer(sublinear_tf=True)\ntfbinicf_vec.fit(count_vec.transform(train_text), train_target)\ntrain_binicf = tfbinicf_vec.transform(count_vec.transform(train_text))\n\nresults = {}\n\ndef validate_results(train_data_vecs, name):\n    skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=13)\n    for i, (train_index, val_index) in enumerate(skf.split(train_text, train_target)):\n        x_train, x_val = train_data_vecs[list(train_index)], train_data_vecs[list(val_index)]\n        y_train, y_val = train_target[train_index], train_target[val_index]\n        classifier = LogisticRegression(C=10, solver='sag', random_state=13)\n        classifier.fit(x_train, y_train)\n        val_preds = classifier.predict_proba(x_val)[:,1]\n        current_results = results.get(name,{'preds': [], 'target': []})\n        current_results['preds'].extend(val_preds)\n        current_results['target'].extend(y_val)\n        results[name] = current_results\n\nvalidate_results(train_rf, 'rf')\nvalidate_results(train_idf, 'idf')\nvalidate_results(train_or, 'or')\nvalidate_results(train_binicf, 'binicf')\nvalidate_results(train_icf, 'icf')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"de0e62e0e987a8f018de220418c6a94782067e8d"},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pylab as plt\n%matplotlib inline\nres = []\nfor k, v in results.items():\n    res.append((k, roc_auc_score(v['target'],np.array(v['preds'])) ,v['preds']))\nres = sorted(res, key= lambda x:-x[1])\ncorrs = np.corrcoef(list(zip(*res))[2])\naccs = list(zip(*res))[1]\nlabels = [f'{x}:{accs[i]:.4f}' for i, x in enumerate(list(zip(*res))[0])]\nfig, ax = plt.subplots(figsize=(10,10)) \nax = sns.heatmap(corrs, \n                 linewidth=0.5, \n                 annot=corrs, \n                 square=True, \n                 ax=ax, \n                 xticklabels=labels,\n                 yticklabels=labels)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7a4c017b8110f3d2efdffb36a7f6bf779251cf02"},"cell_type":"markdown","source":"As you see -- it could be blended, and I hope it will imporove your LB score."},{"metadata":{"trusted":true,"_uuid":"8c9f74d1015b1588c9fbbc1afa7a358c63ae9711"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}