{"cells":[{"metadata":{"_uuid":"53d7babd177ac2aef2ca5e845370a1d865a08bf8","_cell_guid":"95b5ce59-8a90-4298-8001-6c98e17e428f"},"cell_type":"markdown","source":"# Meta Text Features. "},{"metadata":{"_uuid":"cc8d607fd85b24d0eb7c06c2ac7f23770c0830d1","_cell_guid":"b53ed1e6-7515-42b9-9a2c-a563707c37bc"},"cell_type":"markdown","source":"## CNN on title + description -> 't' Score + 'd' Score or 'text' Score"},{"metadata":{"_uuid":"f6c0d1dd9578239390d86e8b29dcbaf5e5e1e263","_cell_guid":"934fface-9697-49a7-a501-8aede72d8260"},"cell_type":"markdown","source":"The model I used for text classification problem is CNN. \n\nYou can check this paper for a detailed explaination.\nhttps://arxiv.org/abs/1408.5882\n\nAlso check this post for an idea of implementation.\nhttp://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/"},{"metadata":{"_uuid":"02d50cee1564085a6ce423386e3a3e9ec551fd23","_cell_guid":"76c4eaaf-cb6a-4b68-ae5d-efc45526349c"},"cell_type":"markdown","source":"I split the train dataset to 3 folds evenly, as the size of test dataset is 1/3 of the size of train dataset. For each fold, it's score is predicted by applying CNN on the other two folds. This is to pretend overfitting. \n\nCheck these for more information. \n\nhttps://www.kdnuggets.com/2017/02/stacking-models-imropved-predictions.html\n\nhttp://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/"},{"metadata":{"_uuid":"9775dfe5fea5aca07c5c4be2a51f232ede254106","_cell_guid":"9b15513a-ed12-4a8b-8059-8bd099e44c57"},"cell_type":"markdown","source":"However, I don't recommended this implementation for real problem because it is time killer (Approx. 20 hours for CV) and have to run over all train data again while new data comes in. \n\nIf any of you know any idea on how to apply this model without training all  over again when new data comes in, PLZ COMMENT BELOW, I'd love to know it.\n\nBelow are some details on time and rmse. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"},"cell_type":"markdown","source":"# Result\nFold 1 Start!\n\nEpoch 1/3\n - 4620s - loss: 0.0581 - mean_squared_error: 0.0581\n\nEpoch 2/3\n - 3155s - loss: 0.0547 - mean_squared_error: 0.0547\n\nEpoch 3/3\n - 3323s - loss: 0.0534 - mean_squared_error: 0.0534\n\nTitle RMSE: 0.234442\n\nEpoch 1/3\n - 6731s - loss: 0.0580 - mean_squared_error: 0.0580\n\nEpoch 2/3\n - 4412s - loss: 0.0546 - mean_squared_error: 0.0546\n\nEpoch 3/3\n - 4420s - loss: 0.0524 - mean_squared_error: 0.0524\n\nDescription RMSE: 0.236288\n\nFold 2 Start!\n\nEpoch 1/3\n - 2436s - loss: 0.0578 - mean_squared_error: 0.0578\n\nEpoch 2/3\n - 2414s - loss: 0.0546 - mean_squared_error: 0.0546\n\nEpoch 3/3\n - 2281s - loss: 0.0534 - mean_squared_error: 0.0534\n\nTitle RMSE: 0.235876\n\nEpoch 1/3\n - 4878s - loss: 0.0600 - mean_squared_error: 0.0600\n\nEpoch 2/3\n - 4345s - loss: 0.0547 - mean_squared_error: 0.0547\n\nEpoch 3/3\n - 9811s - loss: 0.0526 - mean_squared_error: 0.0526\n\nDescription RMSE: 0.235445\n\nFold 3 Start!\n\nEpoch 1/3\n - 3731s - loss: 0.0575 - mean_squared_error: 0.0575\n\nEpoch 2/3\n - 3685s - loss: 0.0545 - mean_squared_error: 0.0545\n\nEpoch 3/3\n - 3645s - loss: 0.0533 - mean_squared_error: 0.0533\n\nTitle RMSE: 0.235326\n\nEpoch 1/3\n - 7152s - loss: 0.0579 - mean_squared_error: 0.0579\n\nEpoch 2/3\n - 6924s - loss: 0.0544 - mean_squared_error: 0.0544\n\nEpoch 3/3\n - 6815s - loss: 0.0523 - mean_squared_error: 0.0523\n\nDescription RMSE: 0.236635\n\nFinish!"},{"metadata":{"_uuid":"304564c0debdb28291096c74739d2f74c7b91d34","_cell_guid":"f44bd8a1-e298-4a0c-8e60-5b76f5df71fc"},"cell_type":"markdown","source":"I will skip the steps on how to generate meta text features. The code for it can be found here. \n\nhttps://github.com/JingqiL/Stacked-models-CNN-XGBoost-/blob/master/Keras%20CNN(title%2Bdescription).ipynb\n\nYou can also first merge two scores of title and description to get an overall score ('text' score) and then feed this score and other features to your ML algorithms. The code of how to merge the score of title and description is in this kernel. (This is a tensorflow version, you can easily transfer it to a Keras version.)\n\nhttps://www.kaggle.com/jingqliu/fasttext-conv2d-with-tf-on-title-and-description/notebook"},{"metadata":{"_uuid":"c8db686477ff1fcb277d869093300b6f48dac24e"},"cell_type":"markdown","source":"#### This version is the result for 'text' score. I have updated the script in github link for both 't'+'d' and 'text'. If you want check the result of 't' + 'd', please go version 6. The rmse is 0.2266."},{"metadata":{"_uuid":"f8c3230cb76e07731c74441d09be77eda48aef88","_cell_guid":"bdcdc387-ab83-46ce-9f09-38099c9ee010"},"cell_type":"markdown","source":"# XGBoost features on meta text + grouped features."},{"metadata":{"_uuid":"14263e5b61ad7fba3bbd3516d17fd1dd86090567","collapsed":true,"_cell_guid":"0062054b-4c20-4ad5-8ae5-073c97c840d6","trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport re\nimport gc\nfrom keras.preprocessing import text, sequence\nimport math\nfrom xgboost.sklearn import XGBRegressor\nimport xgboost as xgb\nfrom sklearn import metrics\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b459d02a92d1105547592fb22fdcadb46e9dd7a1","collapsed":true,"_cell_guid":"174ca03e-d678-41bc-a36b-ab8ddcfd9dae","trusted":false},"cell_type":"code","source":"train = pd.read_csv('../input/readyforuse/trainforuse_mix.csv')\ntest = pd.read_csv('../input/readyforuse/testforuse_mix.csv')\nsubmission = pd.read_csv('../input/avito-demand-prediction/sample_submission.csv')\n\ntrain = train.drop(columns = ['item_id'])\ntrain['date'] = pd.to_datetime(train['activation_date']).dt.weekday.astype('int')\ntrain = train.drop(columns = ['activation_date','image'])\n\ntest = test.drop(columns = ['item_id'])\ntest['date'] = pd.to_datetime(test['activation_date']).dt.weekday.astype('int')\ntest = test.drop(columns = ['activation_date','image'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca0ce61a06d301c527eec21cde36e586963be64e","collapsed":true,"_cell_guid":"a7fc0b73-d4c9-42ac-a2e5-9429b5703ba2","trusted":false},"cell_type":"code","source":"encoding_predictors = ['user_id', 'region', 'city', 'parent_category_name', 'category_name', 'param_1', 'param_2', 'param_3', 'item_seq_number', 'user_type', 'date']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bbea081272cb727cf289afa9f73d2e77b3352b9","collapsed":true,"_cell_guid":"25df4509-b492-4386-9436-345277d4a1c8","trusted":false},"cell_type":"code","source":"for i in encoding_predictors:\n    print(str(i) +': ' + str(len(set(train[i]))))\n    print('Na contains? ' + str(train[i].isnull().values.any()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50e619b4beef277e634285c785a48f76353758bb","collapsed":true,"_cell_guid":"43762c8e-e3b5-4289-9527-95f9380e0501","trusted":false},"cell_type":"code","source":"for i in encoding_predictors:\n    print(str(i) +': ' + str(len(set(test[i]))))\n    print('Na contains? ' + str(test[i].isnull().values.any()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef32909413c5d0892b6efd110be4434d07e2f6ab","collapsed":true,"_cell_guid":"20f9f42b-a7e5-4805-a27a-8842c97af4fd","trusted":false},"cell_type":"code","source":"l_region = dict(zip(list(set(train['region'])),range(1,29)))\nl_parent_category_name = dict(zip(list(set(train['parent_category_name'])),range(1,10)))\nl_category_name = dict(zip(list(set(train['category_name'])),range(1,48)))\nl_user_type = dict(zip(list(set(train['user_type'])),range(1,4)))\nl_date = dict(zip(list(set(train['date'])),range(1,8)))\nl_param1 = dict(zip(list(set(train['param_1'])),range(1,373)))\nl_param2 = dict(zip(list(set(train['param_2'])),range(1,273)))\nl_city = dict(zip(list(set(train['city'])),range(1,1734)))\nl_param3 = dict(zip(list(set(train['param_3'])),range(1,1221)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6df0b59473983a365713204fee59da987790b643","collapsed":true,"_cell_guid":"24caac25-2d07-4bc4-8415-1dceade63ca9","trusted":false},"cell_type":"code","source":"train['region'] = train['region'].replace(l_region)\ntrain['parent_category_name'] = train['parent_category_name'].replace(l_parent_category_name)\ntrain['category_name'] = train['category_name'].replace(l_category_name)\ntrain['user_type'] = train['user_type'].replace(l_user_type)\ntrain['date'] = train['date'].replace(l_date)\ntrain['param_1'] = train['param_1'].replace(l_param1)\ntrain['param_2'] = train['param_2'].replace(l_param2)\ntrain['city'] = train['city'].replace(l_city)\ntrain['param_3'] = train['param_3'].replace(l_param3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"46565b0d0b39038e25f90c5b8f1a17567764c98c","collapsed":true,"_cell_guid":"5b327020-6e97-4a7e-8231-4dfd26b171b2","trusted":false},"cell_type":"code","source":"test['region'] = test['region'].map(l_region)\ntest['parent_category_name'] = test['parent_category_name'].map(l_parent_category_name)\ntest['category_name'] = test['category_name'].map(l_category_name)\ntest['user_type'] = test['user_type'].map(l_user_type)\ntest['date'] = test['date'].map(l_date)\ntest['param_1'] = test['param_1'].map(l_param1)\ntest['param_2'] = test['param_2'].map(l_param2)\ntest['city'] = test['city'].map(l_city)\ntest['param_3'] = test['param_3'].map(l_param3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63583a81c97081b807ae126ae1de64ea938810b6","collapsed":true,"_cell_guid":"e06a4abf-8cce-4234-8623-44a096f21575","trusted":false},"cell_type":"code","source":"def modelfit(alg,dtrain,predictors,useTrainCV = True, cv_folds = 5, early_stopping_rounds = 50):\n    \n    if useTrainCV:\n        xgb_param = alg.get_xgb_params()\n        xgtrain = xgb.DMatrix(dtrain[predictors].values, label = dtrain['deal_probability'].values, feature_names = predictors)\n        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round = alg.get_params()['n_estimators'], nfold = cv_folds, metrics = 'rmse', early_stopping_rounds = early_stopping_rounds)\n        alg.set_params(n_estimators = cvresult.shape[0])\n        print('Best n_estimator = ' + str(cvresult.shape[0]))\n    alg.fit(dtrain[predictors], dtrain['deal_probability'], eval_metric = 'rmse')\n    \n    dtrain_predictions = alg.predict(dtrain[predictors])\n    \n    print('\\nModel Report:')\n    print('RMSE: %f' % math.sqrt(metrics.mean_squared_error(dtrain['deal_probability'].values, dtrain_predictions)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f5e10b6991a94ea5b607e56592331532787fe2d3","collapsed":true,"_cell_guid":"714a9c0c-1e3e-49c5-9c8b-ff19ecf3bdcc","trusted":false},"cell_type":"code","source":"train.loc[:,'image_item_n'] = train.groupby(['image_top_1','item_seq_number']).user_id.transform('nunique')\ntrain.loc[:,'image_city_n'] = train.groupby(['city','image_top_1']).user_id.transform('nunique')\ntrain.loc[:,'image_region_n'] = train.groupby(['region','image_top_1']).user_id.transform('nunique')\ntrain.loc[:,'image_categoryname_n'] = train.groupby(['category_name','image_top_1']).user_id.transform('nunique')\ntrain.loc[:,'image_param2_n'] = train.groupby(['image_top_1','param_2']).user_id.transform('nunique')\ntrain.loc[:,'image_parentcategoryname_n'] = train.groupby(['parent_category_name','image_top_1']).user_id.transform('nunique')\ntrain.loc[:,'image_date_n'] = train.groupby(['image_top_1','date']).user_id.transform('nunique')\ntrain.loc[:,'image_usertype_n'] = train.groupby(['image_top_1','user_type']).user_id.transform('nunique')\ntrain.loc[:,'image_param1_n'] = train.groupby(['image_top_1','param_1']).user_id.transform('nunique')\ntrain.loc[:,'image_param3_n'] = train.groupby(['image_top_1','param_3']).user_id.transform('nunique')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b8c74b27c5c71dd948e65e016e2c617296ed993b","collapsed":true,"_cell_guid":"da372702-a2f6-4060-9373-2b26e32baa43","trusted":false},"cell_type":"code","source":"test.loc[:,'image_item_n'] = test.groupby(['image_top_1','item_seq_number']).user_id.transform('nunique')\ntest.loc[:,'image_city_n'] = test.groupby(['city','image_top_1']).user_id.transform('nunique')\ntest.loc[:,'image_region_n'] = test.groupby(['region','image_top_1']).user_id.transform('nunique')\ntest.loc[:,'image_categoryname_n'] = test.groupby(['category_name','image_top_1']).user_id.transform('nunique')\ntest.loc[:,'image_param2_n'] = test.groupby(['image_top_1','param_2']).user_id.transform('nunique')\ntest.loc[:,'image_parentcategoryname_n'] = test.groupby(['parent_category_name','image_top_1']).user_id.transform('nunique')\ntest.loc[:,'image_date_n'] = test.groupby(['image_top_1','date']).user_id.transform('nunique')\ntest.loc[:,'image_usertype_n'] = test.groupby(['image_top_1','user_type']).user_id.transform('nunique')\ntest.loc[:,'image_param1_n'] = test.groupby(['image_top_1','param_1']).user_id.transform('nunique')\ntest.loc[:,'image_param3_n'] = test.groupby(['image_top_1','param_3']).user_id.transform('nunique')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7478f201d1f1aace718041aea118b4dbc6c1cba6","collapsed":true,"_cell_guid":"c5c7c75e-7315-4f2a-b089-34ebeb8bc382","trusted":false},"cell_type":"code","source":"def clean(string):\n    string = re.sub(r'\\n', ' ', string)\n    string = re.sub(r'\\t', ' ', string)\n    string = re.sub('[\\W]', ' ', string)\n    string = re.sub(r'\\s{2,}', ' ', string.lower())\n    return string\n\ndef find_punc(string):\n    string = re.sub(r'\\s','',string)\n    string = re.findall('[\\W]',string)\n    l = len(string)\n    return l","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"919e05f176140e550f650fe34426df0ca43e1ba8","collapsed":true,"_cell_guid":"5ab81e14-2340-46dd-81ca-de7345e0bfaa","trusted":false},"cell_type":"code","source":"train_t = train['title'].apply(clean)\ntest_t = test['title'].apply(clean)\ntrain_d = train['description'].astype(str).apply(clean)\ntest_d = test['description'].astype(str).apply(clean)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"822f08f79b667945219b96d9d1510d45739a355c","collapsed":true,"_cell_guid":"c24979ee-b24a-4fca-a76e-a08dc105ce9f","trusted":false},"cell_type":"code","source":"train_t_len = []\nfor line in train_t:\n    train_t_len.append(len(line.split()))\n    \ntrain_d_len = []\nfor line in train_d:\n    train_d_len.append(len(line.split()))\n    \ntest_t_len = []\nfor line in test_t:\n    test_t_len.append(len(line.split()))\n    \ntest_d_len = []\nfor line in test_d:\n    test_d_len.append(len(line.split()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cfb14a6bfd8fceac805d90773b36471277ae8163","collapsed":true,"_cell_guid":"914759e2-b314-45c2-b916-78f77838b846","trusted":false},"cell_type":"code","source":"#train['t_n'] = train_t_len\ntrain['t_per'] = np.array(train['title'].apply(find_punc))/np.array(train_t_len)\n#test['t_n'] = test_t_len\ntest['t_per'] = np.array(test['title'].apply(find_punc))/np.array(test_t_len)\ntrain['d_n'] = train_d_len\n#train['d_per'] = np.array(train['description'].astype(str).apply(find_punc))/np.array(train_d_len)\ntest['d_n'] = test_d_len\n#test['d_per'] = np.array(test['description'].astype(str).apply(find_punc))/np.array(test_d_len)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d47760f33a62cfb93d93eb425a1d4bc0c8bdabd","collapsed":true,"_cell_guid":"8b132305-0654-4778-a335-d1ffdda71e40","trusted":false},"cell_type":"code","source":"train = train.drop(columns = ['title','description'])\ntest = test.drop(columns = ['title','description'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93428424f4eaa1b49656efa9ea2e03fc4817869e","collapsed":true,"_cell_guid":"78450dd1-07b0-4538-95cd-085068d21c2a","trusted":false},"cell_type":"code","source":"ready_train = train[['price','image_top_1','param_1','item_seq_number','city','region','parent_category_name','category_name','user_type','date','param_2','deal_probability','param_3','mix'] + list(train.columns[15:])]\nready_test = test[['price','image_top_1','param_1','item_seq_number','city','region','parent_category_name','category_name','user_type','date','param_2','param_3','mix'] + list(test.columns[14:])]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7476390fe88f07690b5c49db72b2ccc5fcc33a9e","collapsed":true,"_cell_guid":"2822b305-b67b-4d76-81e0-0ccd62b535ef","trusted":false},"cell_type":"code","source":"predictors = ready_train.columns[ready_train.columns != 'deal_probability']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75fbfa5e602b125426c3cef5d3142c15ad77eb72","collapsed":true,"_cell_guid":"9a554ccc-8b72-451f-9da9-917f774943d6","trusted":false},"cell_type":"code","source":"len(predictors)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a7038764386a5a1663b63d28b77dd370dee35e35","collapsed":true,"_cell_guid":"0a6cfd7e-8b78-4f4e-8250-b68e67804672","trusted":false},"cell_type":"code","source":"xgb1 = XGBRegressor(objective = 'reg:logistic', learning_rate = 0.1, n_estimators = 1000, max_depth = 5, min_child_weight = 6, gamma = 0, subsample = 0.9, colsample_bytree = 0.7, reg_alpha = 1.4, seed = 2018)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41ae2917ac2508cdebc606c49780d9aaa944fe39","collapsed":true,"_cell_guid":"eba38606-bcc1-4eab-b6da-3a60106e2342","trusted":false},"cell_type":"code","source":"modelfit(xgb1, ready_train, predictors, useTrainCV = False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff45314078ca9d0a5513333ed62577b59d13921b","collapsed":true,"_cell_guid":"f0919bea-de2e-44d1-ae17-116de2f2bc8d","trusted":false},"cell_type":"code","source":"xgb.plot_importance(xgb1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cb5587dc51f9fc50616b178cb230a3351d217f4f","collapsed":true,"_cell_guid":"1aad3ebc-feac-476f-a2e1-9e94abe722ce","trusted":false},"cell_type":"code","source":"pred = xgb1.predict(ready_test[predictors])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6b238e28f0380359acfed24f0c7f7bfac76f0b5","collapsed":true,"_cell_guid":"106eeb05-78ce-4be6-a8b3-9e5d4f410d01","trusted":false},"cell_type":"code","source":"submission['deal_probability'] = pred\nsubmission.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1bda87125a41ffdbdd3fd3cb8c7e44f96c414046","_cell_guid":"5fb4d277-d649-4a8c-8866-e0ffc593d58f"},"cell_type":"markdown","source":"Feature engineering to be continued...... (These features are just for test in order to get a slightly better result than using the original predictors, don't take it seriously.) @.@"},{"metadata":{"_uuid":"7dd6e05f110a0f409e9871a9f23bce181a3a8a19","_cell_guid":"6deec82b-946e-4685-a7df-d27d0dcd2976"},"cell_type":"markdown","source":""}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}