{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Overview\n\nThe purpose of this kernel is to take a look at the data, come up with some insights, and attempt to create a predictive model or two. This notebook is still **very** raw. I will work on it as my very limited time permits, and hope to expend it in the upcoming days and weeks.\n\n## Packages\n\nFirst, let's load a few useful Python packages. This section will keep growing in subsequent versions of this EDA."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport time\nfrom tqdm import tqdm\nimport h2o\n\nimport lightgbm as lgb\n\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import KFold\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB\n\nimport nltk\nfrom nltk.corpus import stopwords\nimport string\n\nfrom scipy.sparse import hstack\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41458c21c61aeef2ef2c6f3d675b9176b1b53d80"},"cell_type":"markdown","source":"This is a kernels-only competition, which means that we can only use tools and data that are available to us in a single Kaggle kernel. The Pyhon libraries that are available to us are the standard Kaggle kernels compute environment. So let's take a look at the data that's available to us: "},{"metadata":{"trusted":true,"_uuid":"746c5a3c8d20adc7cd13f927bf520f938f7073e1"},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fad6f9d51905e8b35c8d36c339ac1f19ca17c820"},"cell_type":"markdown","source":"We see that the input folder contains, in addition to the standard train, test, and sample_submission files, another folder which presumably contains various embeddings. Let's take a look at what embeddings are availabel to us. "},{"metadata":{"trusted":true,"_uuid":"56760e13a98cb4144a3a8934cf437102a8c400e0"},"cell_type":"code","source":"print(os.listdir(\"../input/embeddings\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a44005d69eb359647186f8f72e9744fdb6a4be28"},"cell_type":"markdown","source":"We see that we have access to four different 300-dimensional embeddings. 300-dimensional embeddings are probably the best ones from the standpoint of a single-best-model, but having access to lower dimensional embeddings would have been nice from the prototyping standpoint. We'll get back to the embeddings later, but let's now take a look at the train and test files."},{"metadata":{"trusted":true,"_uuid":"2af70e91c9063219bd5320c3ff492ff6dd1c85c2"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv').fillna(' ')\ntest = pd.read_csv('../input/test.csv').fillna(' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77711fe131e2b3fb71e71a6381393b7d0e777270"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f1755c47bf56c1dedf2703fc1916cbe5a1b8ac02"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6bc03b1bf7a8e2020403e674b362a81dd777b2b5"},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bdf29d2d6ec91ed839ff16a85189360407ff2a4d"},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc3bb9b5f671d737c58c09b2c444f60f0619dd7c"},"cell_type":"markdown","source":"Seems farily straightforward - just ID, text and target firlds. In addition, the train set is very decently sized - 1.3 million records is probably enough for a decent text classifier. \n\nLet's take a look at the targetvariable:\n"},{"metadata":{"trusted":true,"_uuid":"f72f482b966c98b312bbb63d6a6ca123f5ef1f21"},"cell_type":"code","source":"train_target = train['target'].values\n\nnp.unique(train_target)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26eb62061451ea62187fc416b462c1b758f0f674"},"cell_type":"code","source":"train_target.mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a476108d72e163c71d44c7f0b5714b5f54f27d40"},"cell_type":"markdown","source":"That's pretty good: just two classes, but the positive class makes just over 6% of the total. So the target is heavily unbalanced, which is why a metric such as F1 seems appropriate for this kind of problem. "},{"metadata":{"trusted":true,"_uuid":"22c358cbbf6f8257a1518514a517f32b3c13fac1"},"cell_type":"code","source":"eng_stopwords = set(stopwords.words(\"english\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f5cf24bf3261a33472925a43303e38faf87a748"},"cell_type":"markdown","source":"For EDA and later modeling, it might be a good idea to create some metafeatures. This work is partly based on SRK's great EDAs, and [this one](http://www.kaggle.com/sudalairajkumar/simple-feature-engg-notebook-spooky-author) in particular. The metafeatures that we'll create are:\n\n\n* Number of words in the text\n* Number of unique words in the text\n* Number of characters in the text\n* Number of stopwords\n* Number of punctuations\n* Number of upper case words\n* Number of title case words\n* Average length of the words"},{"metadata":{"trusted":true,"_uuid":"58059d55a4ebf812d783b10a4fef3d7b8b268949"},"cell_type":"code","source":"## Number of words in the text ##\ntrain[\"num_words\"] = train[\"question_text\"].apply(lambda x: len(str(x).split()))\ntest[\"num_words\"] = test[\"question_text\"].apply(lambda x: len(str(x).split()))\n\n## Number of unique words in the text ##\ntrain[\"num_unique_words\"] = train[\"question_text\"].apply(lambda x: len(set(str(x).split())))\ntest[\"num_unique_words\"] = test[\"question_text\"].apply(lambda x: len(set(str(x).split())))\n\n## Number of characters in the text ##\ntrain[\"num_chars\"] = train[\"question_text\"].apply(lambda x: len(str(x)))\ntest[\"num_chars\"] = test[\"question_text\"].apply(lambda x: len(str(x)))\n\n## Number of stopwords in the text ##\ntrain[\"num_stopwords\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).lower().split() if w in eng_stopwords]))\ntest[\"num_stopwords\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).lower().split() if w in eng_stopwords]))\n\n## Number of punctuations in the text ##\ntrain[\"num_punctuations\"] =train['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]) )\ntest[\"num_punctuations\"] =test['question_text'].apply(lambda x: len([c for c in str(x) if c in string.punctuation]) )\n\n## Number of title case words in the text ##\ntrain[\"num_words_upper\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\ntest[\"num_words_upper\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.isupper()]))\n\n## Number of title case words in the text ##\ntrain[\"num_words_title\"] = train[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\ntest[\"num_words_title\"] = test[\"question_text\"].apply(lambda x: len([w for w in str(x).split() if w.istitle()]))\n\n## Average length of the words in the text ##\ntrain[\"mean_word_len\"] = train[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))\ntest[\"mean_word_len\"] = test[\"question_text\"].apply(lambda x: np.mean([len(w) for w in str(x).split()]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c416bc48a4e10735c21e3aea819e0f05a880a424"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_words'])\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8f1ad643569958ad2c36990e4f1d9ab2c1b6719"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_unique_words'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc7510393c3d62f75fd06bc64c89138344fd68f2"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_chars'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21424315338743a89e65f70d79e023caa9b51b6c"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_stopwords'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0cecb9d159b833d35276d6508433c737cc99f3ff"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_punctuations'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ae5cfe122f74ef98d62bf9cd5fffd32e67971ed7"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_words_upper'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"223d9d45a38dcb7ff3b1ce13bed17cd19e57b6ae"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['num_words_title'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"14a699bddfa27806d5077e2c0b2eb5e000b2efc3"},"cell_type":"code","source":"plt.figure(figsize=(12,8))\nsns.violinplot(data=train['mean_word_len'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5caf41ac4cd953e6333ca85127666d2427929a02"},"cell_type":"code","source":"eng_features = ['num_words', 'num_unique_words', 'num_chars', \n                'num_stopwords', 'num_punctuations', 'num_words_upper', \n                'num_words_title', 'mean_word_len']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78610ffbbe740c6b7e869292cb4ab42b7c74aa7a"},"cell_type":"code","source":"kf = KFold(n_splits=5, shuffle=True, random_state=43)\ntest_pred = 0\noof_pred = np.zeros([train.shape[0],])\n\nx_test = test[eng_features].values\nfor i, (train_index, val_index) in tqdm(enumerate(kf.split(train))):\n    x_train, x_val = train.loc[train_index][eng_features].values, train.loc[val_index][eng_features].values\n    y_train, y_val = train_target[train_index], train_target[val_index]\n    classifier = LogisticRegression(C= 0.1)\n    classifier.fit(x_train, y_train)\n    val_preds = classifier.predict_proba(x_val)[:,1]\n    preds = classifier.predict_proba(x_test)[:,1]\n    test_pred += 0.2*preds\n    oof_pred[val_index] = val_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50801b38c3049413ee13138d34f3934d4d502c70"},"cell_type":"code","source":"pred_train = (oof_pred > 0.5).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bc1c3ac7676e20e182158cf22d630c0024290d9e"},"cell_type":"code","source":"f1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"10ff7cc20bbf8389132fcfa1d1367eb660afa4db"},"cell_type":"code","source":"pred_train = (oof_pred > 0.12).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d3730ca78361a4dfcf25505b0ab758f485c796be"},"cell_type":"markdown","source":"For our second model we'll use TF-IDF with a logistic regression. The next couple of secontions are based on my [LR with n-grams notebook](https://www.kaggle.com/tunguz/lr-with-words-n-grams-baseline). Firtst, let's embed all the text vectors:"},{"metadata":{"trusted":true,"_uuid":"c5b792f714dcf77c2ac8312aa90f55871f38e511"},"cell_type":"code","source":"train_text = train['question_text']\ntest_text = test['question_text']\nall_text = pd.concat([train_text, test_text])\n\nword_vectorizer = TfidfVectorizer(\n    sublinear_tf=True,\n    strip_accents='unicode',\n    analyzer='word',\n    token_pattern=r'\\w{1,}',\n    stop_words='english',\n    ngram_range=(1, 1),\n    max_features=5000)\nword_vectorizer.fit(all_text)\ntrain_word_features = word_vectorizer.transform(train_text)\ntest_word_features = word_vectorizer.transform(test_text)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3feba195343d625fc8ca3e37dc811d018bc27d09"},"cell_type":"markdown","source":"Now let's see how well a logistic regression trained on these features does:"},{"metadata":{"trusted":true,"_uuid":"e5bfcc006ca924081c10f30f9872272a9a0df802"},"cell_type":"code","source":"kf = KFold(n_splits=5, shuffle=True, random_state=43)\ntest_pred_tf = 0\noof_pred_tf = np.zeros([train.shape[0],])\n\nfor i, (train_index, val_index) in tqdm(enumerate(kf.split(train))):\n    x_train, x_val = train_word_features[train_index,:], train_word_features[val_index,:]\n    y_train, y_val = train_target[train_index], train_target[val_index]\n    classifier = LogisticRegression(class_weight = \"balanced\", C=0.5, solver='sag')\n    classifier.fit(x_train, y_train)\n    val_preds = classifier.predict_proba(x_val)[:,1]\n    preds = classifier.predict_proba(test_word_features)[:,1]\n    test_pred_tf += 0.2*preds\n    oof_pred_tf[val_index] = val_preds\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e2bd290612ddfc4457a5f4d1d60954cbb1f8a128"},"cell_type":"code","source":"pred_train = (oof_pred_tf > 0.8).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f8367b93d7fd7e1ae3c732838901ec7d841bc135"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e29be89e0b8158b802e8e8d7c1f646764f12227"},"cell_type":"code","source":"0.566075663947416","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a88d97f4c75e5783e3fc1b2976f41ef21ec1b145"},"cell_type":"code","source":"pred_train = (0.8*oof_pred_tf+0.2*oof_pred > 0.68).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7b99b64f98bce57352dfe17a91dd820b12762aa6"},"cell_type":"code","source":"0.5705038831309178","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"013bf364e2b27e2f7e79311c33e4284ffe4a4555"},"cell_type":"markdown","source":"The following LightGBM model is based on Peter's [LGB Baseline notebook](https://www.kaggle.com/peterhurford/lgb-baseline):"},{"metadata":{"trusted":true,"_uuid":"0002994a667a6abe93b39c551bd120e6a07bf31b"},"cell_type":"code","source":"import lightgbm as lgb\n\ndef lgb_f1_score(y_hat, data):\n    y_true = data.get_label()\n    y_hat = np.round(y_hat) # scikits f1 doesn't like probabilities\n    return 'f1', f1_score(y_true, y_hat), True\n\nparams = {'learning_rate': 0.05,\n          'application': 'regression',\n          'max_depth': 9,\n          'num_leaves': 100,\n          'verbosity': -1,\n          'metric': 'rmse',\n          'data_random_seed': 3,\n          'bagging_fraction': 0.8,\n          'feature_fraction': 0.4,\n          'nthread': 16,\n          'lambda_l1': 1,\n          'lambda_l2': 1,\n          'num_rounds': 2700,\n          'verbose_eval': 100}\n\n\nkf = KFold(n_splits=5, shuffle=True, random_state=43)\ntest_pred_lgb = 0\noof_pred_lgb = np.zeros([train.shape[0],])\n\nfor i, (train_index, val_index) in tqdm(enumerate(kf.split(train))):\n    x_train, x_val = train_word_features[train_index,:], train_word_features[val_index,:]\n    y_train, y_val = train_target[train_index], train_target[val_index]\n    \n    d_train = lgb.Dataset(x_train, label=y_train)\n    d_valid = lgb.Dataset(x_val, label=y_val)\n\n    num_rounds = 2500\n    model = lgb.train(params,\n                  train_set=d_train,\n                  num_boost_round=num_rounds,\n                  valid_sets=[d_train, d_valid],\n                  valid_names=['train', 'val'],\n                  verbose_eval=0)\n    \n    val_preds = model.predict(x_val)\n    preds = classifier.predict(test_word_features)\n    test_pred_lgb += 0.2*preds\n    oof_pred_lgb[val_index] = val_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d4adfdd4d7a91c8467bec4317b118df582e1814"},"cell_type":"code","source":"pred_train = (oof_pred_lgb > 0.3).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a571cd1b624c2d1c2419cba3af0c8270dc2211d"},"cell_type":"code","source":"pred_train = (0.65*oof_pred_lgb+0.35*oof_pred_tf+0.1*oof_pred > 0.5).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f60dfe7ce557d6d53ddedcdb9594719100cbb38"},"cell_type":"markdown","source":"The following Logistic Regression is based on Premvardhan's [Count Vectorizer notebook](https://www.kaggle.com/premvardhan/quora-insincere-question-classification):"},{"metadata":{"trusted":true,"_uuid":"51b23339c71a594f458404659615dca2782483f5"},"cell_type":"code","source":"# Train Vectorizor\nfrom sklearn.feature_extraction.text import CountVectorizer \n\nbow = CountVectorizer()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d2552874312556fdd1cc971106bef7e22edeeea3"},"cell_type":"code","source":"kf = KFold(n_splits=5, shuffle=True, random_state=43)\ntest_pred_cv = 0\noof_pred_cv = np.zeros([train.shape[0],])\n\n\nfor i, (train_index, val_index) in tqdm(enumerate(kf.split(train))):\n    x_train, x_val = train.loc[train_index]['question_text'].values, train.loc[val_index]['question_text'].values\n    y_train, y_val = train_target[train_index], train_target[val_index]\n    x_test = test['question_text'].values\n    \n    bow = CountVectorizer()\n    x_train = bow.fit_transform(x_train)\n    x_val = bow.transform(x_val)\n    x_test = bow.transform(x_test)\n\n    classifier = LogisticRegression(penalty = \"l1\", C = 1.25, class_weight = \"balanced\")\n    \n    classifier.fit(x_train, y_train)\n    val_preds = classifier.predict_proba(x_val)[:,1]\n    preds = classifier.predict_proba(x_test)[:,1]\n    test_pred_cv += 0.2*preds\n    oof_pred_cv[val_index] = val_preds","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"68500c1f4086472f67780f91ba47f1c13e713995"},"cell_type":"markdown","source":"The following classifiers are inspired by dust's [Naive Bayes notebook](https://www.kaggle.com/stardust0/naive-bayes-and-logistic-regression-baseline):"},{"metadata":{"trusted":true,"_uuid":"532a853aa01de9d2795b15647e839efc9b3cd6ae"},"cell_type":"code","source":"kf = KFold(n_splits=5, shuffle=True, random_state=43)\ntest_pred_cv_2 = 0\noof_pred_cv_2 = np.zeros([train.shape[0],])\ntest_pred_cv_3 = 0\noof_pred_cv_3 = np.zeros([train.shape[0],])\n\n\nfor i, (train_index, val_index) in tqdm(enumerate(kf.split(train))):\n    x_train, x_val = train.loc[train_index]['question_text'].values, train.loc[val_index]['question_text'].values\n    y_train, y_val = train_target[train_index], train_target[val_index]\n    x_test = test['question_text'].values\n    \n    bow = CountVectorizer()\n    x_train = bow.fit_transform(x_train)\n    x_val = bow.transform(x_val)\n    x_test = bow.transform(x_test)\n    \n    classifier2 = MultinomialNB()\n    classifier3 = BernoulliNB()\n    \n    classifier2.fit(x_train, y_train)\n    val_preds = classifier2.predict_proba(x_val)[:,1]\n    preds = classifier2.predict_proba(x_test)[:,1]\n    test_pred_cv_2 += 0.2*preds\n    oof_pred_cv_2[val_index] = val_preds\n    \n    classifier3.fit(x_train, y_train)\n    val_preds = classifier3.predict_proba(x_val)[:,1]\n    preds = classifier3.predict_proba(x_test)[:,1]\n    test_pred_cv_3 += 0.2*preds\n    oof_pred_cv_3[val_index] = val_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c1c4839fec39da8b134f3e0bd48d858d184ed09f"},"cell_type":"code","source":"pred_train = (oof_pred_cv > 0.75).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e272dbc4b62a2fa86697b91e4dc27d319738b35"},"cell_type":"code","source":"pred_train = (oof_pred_cv_2 > 0.7).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f2fe29fea3f7b77ad17aa4b822ef8ea1b43618c2"},"cell_type":"code","source":"pred_train = (oof_pred_cv_3 > 0.7).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3de26da34190ea9a34c82ad88a0a21a3d8e6698"},"cell_type":"code","source":"pred_train = (0.7*oof_pred_cv+0.2*oof_pred_cv_2+0.1*oof_pred_cv_3 > 0.7).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"325667baf436d5a219ceed10ed13cb1af6bf108f"},"cell_type":"code","source":"pred_train = (0.63*(0.7*oof_pred_cv+0.2*oof_pred_cv_2+0.1*oof_pred_cv_3) +0.37*(0.65*oof_pred_lgb+0.35*oof_pred_tf+0.1*oof_pred)> 0.59).astype(np.int)\nf1_score(train_target, pred_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a71bf8e02b6a3ce39e52aca75af63ea5efded15"},"cell_type":"code","source":"stack_train = np.hstack((oof_pred.reshape(-1,1), oof_pred_tf.reshape(-1,1), oof_pred_lgb.reshape(-1,1), \n                         oof_pred_cv_3.reshape(-1,1), oof_pred_cv_2.reshape(-1,1), oof_pred_cv.reshape(-1,1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04db197273bb1eb232fd578ae427654c8fe8faa2"},"cell_type":"code","source":"stack_test = np.hstack((test_pred.reshape(-1,1), test_pred_tf.reshape(-1,1), test_pred_lgb.reshape(-1,1), \n                         test_pred_cv_3.reshape(-1,1), test_pred_cv_2.reshape(-1,1), test_pred_cv.reshape(-1,1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61f701943e4e152a0041704826eb48ed021157cb"},"cell_type":"code","source":"stack_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"532f565f5c1b0e6d0dfbddef40847ecab2ab91a9"},"cell_type":"code","source":"stack_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d00c2dc0e6c51cc0b8d4b89e1d7b57bbbf6337d"},"cell_type":"code","source":"kf = KFold(n_splits=5, shuffle=True, random_state=43)\ntest_pred_stack = 0\noof_pred_stack = np.zeros([train.shape[0],])\n\nfor i, (train_index, val_index) in tqdm(enumerate(kf.split(train))):\n    x_train, x_val = stack_train[train_index,:], stack_train[val_index,:]\n    y_train, y_val = train_target[train_index], train_target[val_index]\n    classifier = LogisticRegression(class_weight = \"balanced\", C=0.5, solver='sag')\n    classifier.fit(x_train, y_train)\n    val_preds = classifier.predict_proba(x_val)[:,1]\n    preds = classifier.predict_proba(stack_test)[:,1]\n    test_pred_stack += 0.2*preds\n    oof_pred_stack[val_index] = val_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a3b41b8018755cab5c3c47682e0fe06507765ff5"},"cell_type":"code","source":"score = 0\nthresh = .5\nfor i in np.arange(0.1, 0.951, 0.01):\n    temp_score = f1_score(train_target, (oof_pred_stack > i))\n    if(temp_score > score):\n        score = temp_score\n        thresh = i\n\nprint(\"CV: {}, Threshold: {}\".format(score, thresh))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4255cb5cb3ec09c23c7e47c0101eaa47e0b59fd9"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"60f2e8583cf77110585641e83093b43cee9e9aae"},"cell_type":"code","source":"0.6207799320845656","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2231a1f454525c3b97a837fd04bcd464d66ef7af"},"cell_type":"markdown","source":"Now we'll train on the full set and make predictions based on that:"},{"metadata":{"trusted":true,"_uuid":"fbc563deb2a06d6db182f4643c3ee8eb68d43f67"},"cell_type":"code","source":"pred_test = ( test_pred_stack> thresh).astype(np.int)\nsubmission = pd.DataFrame.from_dict({'qid': test['qid']})\nsubmission['prediction'] = pred_test\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2ec1994b2e44fd068c059937396e143c28c36a66"},"cell_type":"markdown","source":"To be continued ..."},{"metadata":{"trusted":true,"_uuid":"8d541ab84ba32bd692f9779727a7914feeeb0bc1"},"cell_type":"code","source":"1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6fa0ebe87bd3d31c23d45dc281390cefe47651ec"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}