{"cells":[{"metadata":{"_cell_guid":"fd91185c-9038-42a8-a2ac-740c131bbad6","_uuid":"7c7ef7f9b0f3f029b560322f06ba90cebe7a7c23"},"cell_type":"markdown","source":"# Overview of brilliant kernels "},{"metadata":{"_cell_guid":"76f3b08f-d7f6-4433-ab11-c52817f68379","_uuid":"e3fb419ce2b94224d68493ebbc27a461bbd76cdb"},"cell_type":"markdown","source":"## 1.data preprocessing & FE\n### downsample of train data"},{"metadata":{"_cell_guid":"e0787824-0b12-482e-b329-87f6d15767c7","_uuid":"3b7a9c3fab26b6a61bb43a6a6899ec3eae3f26d0"},"cell_type":"markdown","source":"\n* one-hot encoding/mean encoding for app,channel\n* time pattern analysis\n* keep all positive observations (observations with is_attributed = 1),randomly selected 5 million observations from null observations(observations with is_attributed = 1)"},{"metadata":{"_cell_guid":"26d3b339-cf37-459e-80d8-0f27119c98fa","_uuid":"09e5406124233740f8362b912788048b4aa67292"},"cell_type":"markdown","source":"### data observations\n* most frequent hours in test data:[4,5,9,10,13,14]\n* least frequent hours in test data:[6,11,15]"},{"metadata":{"_cell_guid":"81070c06-9087-4c07-9f01-e41fdba93433","_uuid":"805f6c368fb10cc763b44a40cf6bbf9904f5d67b"},"cell_type":"markdown","source":"### new created features\n* treat **ip,device,os** as an unique user instance\n* find discriminative features from combination of (ip,device,os) and app, for they are independent of the ad publishers"},{"metadata":{"_cell_guid":"f114f350-1303-4901-b8df-acb1502ab77e","_uuid":"8762477aec4ddb8e09a48c362c6ee56a429d31ad"},"cell_type":"markdown","source":"### integrated features\n* ip-wday-hour\n* ip-wday-hour-os\n* ip-wday-hour-app\n* ip-wday-hour-app-os\n* app-wday-hour"},{"metadata":{"_cell_guid":"f2badba3-a9b2-47d8-86a9-610ff3e043f8","_uuid":"cba8f2656c99e81e3c8895b6df8bda1ee5bb6356"},"cell_type":"markdown","source":"### experiments\n* most previous kernels incorporate FE that combina multi given features, but intuitively, some of them make no sense,so i propose to use simplified given features to train the classification model.\n* the ip pattern analysis indicates that **ip** feature could be divided into 4 periods, so I just chose training set that has ip>=126420,leading a local 0.979 score."},{"metadata":{"_uuid":"e3dcf09c581fdcdf133081461f9dd5981c6bfd80","_cell_guid":"fe5c19cd-8c43-4cd8-884b-7be907262c6f","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport time\nimport numpy as np\nfrom sklearn.cross_validation import train_test_split\nimport xgboost as xgb\n\ndef dataPreProcessTime(df):\n    df['click_time'] = pd.to_datetime(df['click_time']).dt.date\n    df['click_time'] = df['click_time'].apply(lambda x: x.strftime('%Y%m%d')).astype(int)\n    \n    return df\n\nstart_time = time.time()\n\ntrain = pd.read_csv('../input/train.csv', skiprows=160000000, nrows=40000000)\ntest = pd.read_csv('../input/test.csv')\ntrain.columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'attributed_time', 'is_attributed']\n\nprint('[{}] Finished loading data'.format(time.time() - start_time))\n\ntrain = dataPreProcessTime(train)\ntest = dataPreProcessTime(test)\n\ntrain=train[train.ip>=126420]\n\ny = train['is_attributed']\ntrain.drop(['is_attributed', 'attributed_time'], axis=1, inplace=True)\n\nsub = pd.DataFrame()\nsub['click_id'] = test['click_id']\ntest.drop('click_id', axis=1, inplace=True)\n\n\n# train.drop('click_time',axis=1,inplace=True)\n# train.drop('os',axis=1,inplace=True)\n# train.drop('ip',axis=1,inplace=True)\n# train.drop('app',axis=1,inplace=True)\n\n\nprint('[{}] Start XGBoost Training'.format(time.time() - start_time))\n\nparams = {'eta': 0.1, \n          'max_depth': 4, \n          'subsample': 0.9, \n          'colsample_bytree': 0.7, \n          'colsample_bylevel':0.7,\n          'min_child_weight':100,\n          'alpha':4,\n          'objective': 'binary:logistic', \n          'eval_metric': 'auc', \n          'random_state': 99, \n          'silent': True,\n          'subsample':0.5}\n          \nx1, x2, y1, y2 = train_test_split(train, y, test_size=0.1, random_state=99)\n\nwatchlist = [(xgb.DMatrix(x1, y1), 'train'), (xgb.DMatrix(x2, y2), 'valid')]\nmodel = xgb.train(params, xgb.DMatrix(x1, y1), 260, watchlist, maximize=True, verbose_eval=10)\n\nprint('[{}] Finish XGBoost Training'.format(time.time() - start_time))\n\nsub['is_attributed'] = model.predict(xgb.DMatrix(test), ntree_limit=model.best_ntree_limit)\nsub.to_csv('syp_xgb_sub_ip_le.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a8f3c417-958d-47c2-a4a1-a0ac59f16113","_uuid":"474cecb65dff900ac4027316ee8e5c1949dc64e0"},"cell_type":"markdown","source":"## 3.params tuning for xgb\n"},{"metadata":{"_cell_guid":"dcddd7e0-3035-4fa0-a9e5-6140ee68f2b8","_uuid":"641582b6372ada72db320a1f7584cd088e970114"},"cell_type":"markdown","source":"### tips for tuning params"},{"metadata":{"_cell_guid":"eb05c1b7-dceb-44b5-8fac-b1f0c691cbda","_uuid":"59659e4eb0579035ecb46be5debc30c346c8e2d9"},"cell_type":"markdown","source":"**max_depth** : \n*    higher tree depth makes more complicated models, also making it possible to learn more local and concrete samples. \n*    it often be set during 3~10\n*    the cross validation could help to tune tree depth\n\n**filter features from fea importance generated from xgb**\n\n**little learning rate with larger n_estimators that could get from CV**\n\n**subsample,colsample_bytree,colsample_bylevel could be set in 0.3-0.8, random sample is better in most conditions**"},{"metadata":{"_cell_guid":"f5db330d-d4cb-475c-b6c8-3a783cfb496d","_uuid":"a86ad6b14b3bdf38dea17bc381001f461e1edf6c"},"cell_type":"markdown","source":"## 4.blending&stacking\n* with weight\n* logit combination\n* average \n* rank"},{"metadata":{"_cell_guid":"895c2abb-41f7-442e-b931-6dd5537da8df","_uuid":"f2f467f225e32fda7211eb6c715cbabdc74c29f7"},"cell_type":"markdown","source":"## references\n i wrote this gratefully with reference to most highly upvoted kernes"},{"metadata":{"collapsed":true,"_uuid":"549b5e02ab7e9fde1b4def8a43838a48e9649bab","_cell_guid":"ddd5003b-4af8-42d6-b07b-1b00a5ce27e0","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}