{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Lets try to cook the text meta-features to find any valuable information. \n\n**Recipe:** \n* First calculate some basic stats like text length, number of letters, numbers and other characters in text in both sincere in insincere questions.  \n* Then  'borrow' some ideas from this amazing kernel https://www.kaggle.com/thebrownviking20/analyzing-quora-for-the-insinceres# to calculate text quality and readability indices. \n* Afterwards, check mean similarites using student's T-Test \n* Finally put all incredients into a LightGBM and cook for a few iterations\n\n**Result: Food not eatable**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#Import needed libraries\nimport itertools\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import train_test_split\nimport lightgbm as lgb\nimport textstat\nfrom scipy import stats\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"568712643dd66a2044f30ce666875e2a19cfdc48"},"cell_type":"code","source":"#Import data\nprint('Importing data...')\ndf_train = pd.read_csv(\"../input/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22de7b494772646532318a18d98cf04ac4f803b8"},"cell_type":"code","source":"#meta features\ndf_train['length'] = df_train['question_text'].str.len()\n\ndf_train['numbers'] = df_train['question_text'].apply(lambda x: len([s for s in x if s.isdigit()]))\ndf_train['words'] = df_train['question_text'].apply(lambda x: len([s for s in x if s.isalpha()]))\ndf_train['spaces'] = df_train['question_text'].apply(lambda x: len([s for s in x if s.isspace()]))\ndf_train['other_chars'] = df_train['length'] - df_train['numbers'] - df_train['words'] - df_train['spaces']\n\ndf_train['numbers_ratio'] = df_train['numbers'] / df_train['length']\ndf_train['words_ratio'] = df_train['words'] / df_train['length']\ndf_train['spaces_ratio'] = df_train['spaces'] / df_train['length']\ndf_train['other_chars_ratio'] = df_train['other_chars'] / df_train['length']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a15bc0c2fab72b702ea43e8cd974a74a75aadaa"},"cell_type":"code","source":"#Sentence readability\nprint('Flesch index..')\ndf_train['flesch'] = df_train['question_text'].apply(lambda x: textstat.flesch_reading_ease(x))\nprint('gunning index..')\ndf_train['gunning'] = df_train['question_text'].apply(lambda x: textstat.gunning_fog(x))\nprint('smog index..')\ndf_train['smog'] = df_train['question_text'].apply(lambda x: textstat.smog_index(x))\nprint('auto index..')\ndf_train['auto'] = df_train['question_text'].apply(lambda x: textstat.automated_readability_index(x))\nprint('coleman index..')\ndf_train['coleman'] = df_train['question_text'].apply(lambda x: textstat.coleman_liau_index(x))\nprint('linear index..')\ndf_train['linsear'] = df_train['question_text'].apply(lambda x: textstat.linsear_write_formula(x))\nprint('dale index..')\ndf_train['dale'] = df_train['question_text'].apply(lambda x: textstat.dale_chall_readability_score(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9bd4540b4db2241c558b1ad50aee9f053ce10b85"},"cell_type":"code","source":"#Separate in sincere and insicere dataframes \nsincere_df = df_train[df_train['target']==0]\ninsincere_df = df_train[df_train['target']==1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0d3133537245bb9bc970a549a670078849f50dc"},"cell_type":"code","source":"#Make sure we have enough samples for statistical significance\nprint('Sentences in sincere dataframe:', sincere_df.shape[0])\nprint('Sentences in insincere dataframe:', insincere_df.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61308dfadc6e026012d6779453d4298ae04cf3b0"},"cell_type":"markdown","source":"## Two sample T-Tests of mean similarity\n\nIn a student's t-test the null hypothesis is that the two means under consideration are statistically equal. If the p-value of the test is less than a significance threshold this null hypothesis is rejected. For example if p-value = 0.002 we can reject the null hypothesis at 5% significance level. \n\nCaution: That DOES not mean that we automatically accept the non-null hypothesis of mean inequality - we only reject the null hypothesis."},{"metadata":{"trusted":true,"_uuid":"472f784e7ff01af0d9b3a6603bcfc1fbe80b50ea"},"cell_type":"code","source":"#Perform students t-test\ncheck_columns = ['length', 'numbers', 'words', 'spaces', 'other_chars', 'numbers_ratio', \n                 'words_ratio', 'spaces_ratio','other_chars_ratio', 'flesch', 'gunning', \n                 'smog', 'auto', 'coleman', 'linsear', 'dale']\n\nfor col in check_columns:\n    t2, p2 = stats.ttest_ind(sincere_df[col],insincere_df[col])\n    print('t-stat:', t2, '. p-value:', p2)\n    if p2<0.05: \n        print('For feature', col, 'means are: DIFFERENT')\n    elif p2>=0.05: \n        print('For feature', col, 'means are: SAME')\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2e8b472693b89126cd0785278981c06edf736b4"},"cell_type":"markdown","source":"Seems like in most case there is a statistical difference of the means between sincere and insincere questions. I wonder if that will provide any actual value in a Machine Learning model. Let's try. "},{"metadata":{"trusted":true,"_uuid":"ac6a4ce0e4877ef2894a0a65dabc32fd3a14cad1"},"cell_type":"code","source":"#Split in train and test\ntrain, valid = train_test_split(df_train, test_size=0.15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"83f4ed0f3f4e6c007b62e53ad7001df4a7253d5a"},"cell_type":"code","source":"train_x = train.drop(['qid', 'question_text', 'target', 'words', 'length', \n                      'words_ratio', 'spaces'], axis = 1)\ntrain_y = train['target']\n\nvalid_x = valid.drop(['qid', 'question_text', 'target', 'words', 'length', \n                      'words_ratio', 'spaces'], axis = 1)\nvalid_y = valid['target']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1abc9e7cc6631787d7e297ce2c05202cd4c9b70d"},"cell_type":"code","source":"#LGB model\nlgb_train = lgb.Dataset(train_x, train_y)\nlgb_valid = lgb.Dataset(valid_x, valid_y)\n\n# Specify hyper-parameters as a dict\nparams = {\n    'task': 'train',\n    'boosting_type': 'gbdt',\n    'objective': 'binary',\n    'metric': 'auc',\n    'num_leaves': 18,\n    'max_depth': 4,\n    'learning_rate': 0.05,\n    #'feature_fraction': 0.95,\n    #'bagging_fraction': 0.8,\n    #'bagging_freq': 5,\n    #'reg_alpha': 0.1,\n    #'reg_lambda': 0.1,\n    'is_unbalance': True,\n    'num_class': 1,\n    #'scale_pos_weight': 3.2,\n    'verbose': 1,\n}\n\nnum_iter = 500\n\n# Train LightGBM model\nprint('Start training...')\ngbm = lgb.train(params,\n                lgb_train,\n                num_boost_round=num_iter,\n                valid_sets= lgb_valid,\n                early_stopping_rounds=40,\n                verbose_eval=20\n                )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"642dd2c6f5642972f198b70084d00e66a37e5ef5"},"cell_type":"code","source":"# Plot Importances\nprint('Plot feature importances...')\nimportances = gbm.feature_importance(importance_type='gain')  # importance_type='split'\nmodel_columns = pd.DataFrame(train_x.columns, columns=['features'])\nfeat_imp = model_columns.copy()\nfeat_imp['importance'] = importances\nfeat_imp = feat_imp.sort_values(by='importance', ascending=False)\nfeat_imp.reset_index(inplace=True)\n\nplt.figure()\nplt.barh(np.arange(feat_imp.shape[0] - 1, -1, -1), feat_imp.importance)\nplt.yticks(np.arange(feat_imp.shape[0] - 1, -1, -1), (feat_imp.features))\nplt.title(\"Feature Importances\")\nplt.ylabel('Feature')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5a9f388029342ecd5167a1f3b6b6e039af0987b8"},"cell_type":"code","source":"pred_lgb = gbm.predict(valid_x,\n                       num_iteration=40\n                       )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c221b311c6bd2cc19935200bff00bd817f681701"},"cell_type":"code","source":"#Sensitivity analysis\nsteps = np.arange(0.1,0.7, 0.01)\nvalidation_pred = []\nfor i in steps:\n    valid_pred_01_lstm = np.where(pred_lgb > i, 1, 0)\n    valid_pred_01_lstm = [int(item) for item in valid_pred_01_lstm]\n    f1_quora = f1_score(valid_y, valid_pred_01_lstm)\n    validation_pred.append(f1_quora)\n\nplt.figure()\nplt.plot(steps, validation_pred)\nplt.grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae5ec763116c716257bc53b91a475b7d87309e48"},"cell_type":"markdown","source":"## Verdict: \nSeems like meta-features have little explanatory power on their own. Maybe if we put them in an ensemble they could capture some different signal and thus contribute positively to our model but I doubt even that..."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}