{"cells":[{"metadata":{"_uuid":"a80888df3d291aafac5f76d5070ebd6ff80a533f"},"cell_type":"markdown","source":"# Xgboost 하이퍼 파라미터 튜닝"},{"metadata":{"_uuid":"55f79f1c3e56e5d55601e2a8f0bda47ffafd6daf"},"cell_type":"markdown","source":"다음 글을 참고 및 번역했습니다. <br/>\n[Complete Guide to Parameter Tuning in XGBoost (with codes in Python)](https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/)"},{"metadata":{"_uuid":"ba09282bd59616d198cabfc8ebec4a1da293a609"},"cell_type":"markdown","source":"Table of Contents\n- Xgboost 장점\n- Xgboost Parameter 이해하기\n- Parameter Tuning 하기\n"},{"metadata":{"_uuid":"1be45ce414f7793aad3150beb772ab8c48b2bfe9"},"cell_type":"markdown","source":"## 1. Xgboost 장점\n\nXgboost는 기존 GBDT 모델에 비해서 다음 기능이 있다.\n\n- 정규화(Regularization)\n- 병렬 처리\n- 고수준의 유연성\n- 결측치 처리\n- Tree Pruning\n- 내장 Cross Validation\n- 기존 모델에 이어서 재학습할 수 있음"},{"metadata":{"_uuid":"ee25f5dd6ef4ba11c7708b97dae6107798155a23"},"cell_type":"markdown","source":"## 2. Xgboost Hyperparameter\n\nXgboost는 다음과 같은 Hyperparameter가 있습니다."},{"metadata":{"_uuid":"de89c7f604913e5612bb752b75ea4b3f462aaafd"},"cell_type":"markdown","source":"- Parameter 종류\n    - General Parameter: 전체 기능을 가이드\n    - Boost Parameter: 각각의 step에서 booster 가이드\n    - Learning Task Parameter: 최적화 수행 가이드"},{"metadata":{"_uuid":"3b78ad76f7a3e3252c889f14d956b291bb1f87ae"},"cell_type":"markdown","source":"1. General Parameter\n    - booster: tree 기반 모델 / 선형 모델\n    - silent: 메세지 조절\n    - nthread: 병렬 처리 조절"},{"metadata":{"_uuid":"595a5b8212b3712445b76210e4e8f8daf2b53cb8"},"cell_type":"markdown","source":"2. Boost Parameter\n    - eta: Learning rate(일반적으로 0.01 - 0.2)\n    - min_child_weight: min_child_weight를 기준으로 추가 분기 결정(크면 Underfitting)\n    - max_depth: Tree 깊이 수\n    - max_leaf_node: 하나의 트리에서 node 개수\n    - gamma: split 하기 위한 최소의 loss 감소 정의\n    - subsample: 데이터 중 샘플링(0.5 - 1)\n    - colsample_bytree: column 중 sampling(0.5 - 1)\n    - colsample_bylevel: 각 level마다 샘플링 비율\n    - lambda: L2 nrom\n    - alpha: L1 norm\n    - scale_pos_weight: positive, negative weight 지정\n    - 기타 등"},{"metadata":{"_uuid":"47bc660b1da2c3384d62d4f66eebad3c35d52365"},"cell_type":"markdown","source":"3. Learning Task Parameter\n    - object: 목적함수 종류\n        - binary:logistic(이진 분류)\n        - multi:softmax(다중 분류)\n        - multi-softprob(다중 확률)    \n    - eval_metric: 평가 지표\n        - rmse – root mean square error\n        - mae – mean absolute error\n        - logloss – negative log-likelihood\n        - error – Binary classification error rate (0.5 threshold)\n        - merror – Multiclass classification error rate\n        - mlogloss – Multiclass logloss\n        - auc: Area under the curve\n    - seed\n"},{"metadata":{"_uuid":"71b137f57f3fd9f7d6ca934a1afeb0dc5b283733"},"cell_type":"markdown","source":"> ## 3. 하이퍼파라미터 튜닝"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e74896ecaec6dd5506db80875604b48d6f2dc76"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport xgboost as xgb\nfrom xgboost.sklearn import XGBClassifier\nfrom sklearn import metrics   #Additional scklearn functions\nfrom sklearn.model_selection import GridSearchCV   #Perforing grid search\n\nimport matplotlib.pylab as plt\n%matplotlib inline\nfrom matplotlib.pylab import rcParams\nrcParams['figure.figsize'] = 12, 4","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ec99348f4ed5f9dcb6b696f2e3f7c9580222ec98"},"cell_type":"markdown","source":"### 데이터 정제하기\n\n여기에서는 Hyperparameter Tuning이 목적이므로 전처리를 자세히 하지는 않겠습니다."},{"metadata":{"trusted":true,"_uuid":"47ffeb7f0ce751d9ea587227d3ba70ebf4baf667"},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b476dba8ce20474be42951ab3a394c5767fab4b4"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33d060147c8d79262248d53f75783a72fe58cd4e"},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ee22e91c990416feb98440e02a296856faf3ddb5"},"cell_type":"code","source":"# 처리하기 복잡한 Column은 제거\ndel train['Ticket']; del test['Ticket']\ndel train['Cabin']; del test['Cabin']\ndel train['Name']; del test['Name']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f95fe6d33567626b8fe87d7a2a0f9722486dae3c"},"cell_type":"code","source":"# train, test에 다른 Category 존재 가능\ntest.insert(loc=1, column='Survived', value=0)\ntotal = pd.concat([train, test], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61bc1c929c2882f0a2d1b1b5d6d415fd48286da4"},"cell_type":"code","source":"# One hot encoding\nsex = pd.get_dummies(total['Sex'])\nembarked = pd.get_dummies(total['Embarked'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37e644a7562d8b634b9adbb951d9de7b2739e5db"},"cell_type":"code","source":"# 기존 컬럼 제거\ndel total['Sex']\ndel total['Embarked']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af8c6a1b0c5b6184e8862c930c216e72977e32bf"},"cell_type":"code","source":"total = pd.concat([total, sex, embarked], axis=1)\ntotal['Family'] = total['Parch'] + total['SibSp']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d81c7122ba543edd9b51a55995f274d2880cb1ec"},"cell_type":"code","source":"# one hot 컬럼이 있는 train, test \ntrain = total[0:len(train)]\ntest = total[len(train):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f8ed27981e8ec4a5b8c0ae9873dfcd51999a485"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"60c63bb9bc3d1f6e99c900034318e6b5ac078ab7"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12c8dcb59ee3bba8b5ec70699f48e791c5869428"},"cell_type":"markdown","source":"## 예측 모델 함수 생성"},{"metadata":{"trusted":true,"_uuid":"12637745259ffcac4b75f7ac5b09675759de2824"},"cell_type":"code","source":"target = 'Survived'\nIDcol = 'PassengerId'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c32049a0fb5e8006206d19f067a575f7f58286de"},"cell_type":"code","source":"def modelfit(alg, dtrain, predictors,useTrainCV=True, cv_folds=5, early_stopping_rounds=100):\n   \n    # get new n_estimator\n    if useTrainCV:\n        xgb_param = alg.get_xgb_params()\n        xgtrain = xgb.DMatrix(dtrain[predictors].values, label=dtrain[target].values)\n        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n                          metrics='error', early_stopping_rounds=early_stopping_rounds)\n        alg.set_params(n_estimators=cvresult.shape[0])\n        print(alg)\n    \n    # Fit the algorithm on the data\n    alg.fit(dtrain[predictors], dtrain['Survived'], eval_metric='error')\n        \n    #Predict training set:\n    dtrain_predictions = alg.predict(dtrain[predictors])\n    dtrain_predprob = alg.predict_proba(dtrain[predictors])[:,1]\n        \n    #Print model report:\n    print(\"\\nModel Report\")\n    print(\"Training Accuracy : %.4g\" % metrics.accuracy_score(dtrain['Survived'].values, dtrain_predictions))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0121637140462c416a9b5df15d76a61592186d4b"},"cell_type":"markdown","source":"## 3. 일반적인 Hyperparameter 튜닝 방법\n\n1. high learning rate(0.05 - 0.3)를 선택하고 이 학습률에 맞는 tree 개수를 선정한다.\n2. tree-specific parameter를 수정한다.\n    - max_depth, min_child_weight, gamma, subsample, colsample_bytree\n3. regularization parameter를 수정한다.\n4. 학습률을 낮추고 다시 반복한다."},{"metadata":{"_uuid":"6b069825d14fe15894efd4c1bf8ae3989ebebc20"},"cell_type":"markdown","source":"## 3-1. Learning rate와  estimator 수를 고정한다."},{"metadata":{"_uuid":"626b4a783365079211b75fe995725cf295b8fcbb"},"cell_type":"markdown","source":"초기값은 다음과 같이 선정한다.\n\n1. max_depth = 5: 보통 4-6 를 시작점으로 한다.\n\n2. min_child_weight = 1 : 향후에 튜닝할 것이다.\n\n3. gamma = 0 :  0.1 - 0.2로 시작해도 된다. 그런데 어짜피 튜닝할 것이다.\n\n4. subsample, colsample_bytree = 0.8 : 보통 0.5 - 0.9로 시작한다.\n\n5. scale_pos_weight = 1: Because of high class imbalance.\n"},{"metadata":{"trusted":true,"_uuid":"e02445276937153d8c839eb6481a78c2833e2a9d"},"cell_type":"code","source":"predictors = [x for x in train.columns if x not in [target, IDcol]]\nxgb1 = XGBClassifier(\n    learning_rate =0.1,\n    n_estimators=1000,\n    max_depth=5,\n    min_child_weight=1,\n    gamma=0,\n    subsample=0.8,\n    colsample_bytree=0.8,\n    objective= 'binary:logistic',\n    nthread=-1,\n    scale_pos_weight=1,\n    seed=2019\n)\nmodelfit(xgb1, train, predictors)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"53f3e2399e73f3bc9d68706e2d4f28ef3928f781"},"cell_type":"markdown","source":"## 3-2. max_depth와 min_child_weight를 튜닝한다."},{"metadata":{"trusted":true,"_uuid":"099f77a66ac6489a10a1d317e4a7a619d204117c"},"cell_type":"code","source":"param_test1 = {\n 'max_depth':range(3,10,3),\n 'min_child_weight':range(1,6,2)\n}\ngsearch1 = GridSearchCV(estimator = XGBClassifier(learning_rate=0.1, \n                                                  n_estimators=1000, \n                                                  max_depth=5, \n                                                  min_child_weight=1, \n                                                  gamma=0, \n                                                  subsample=0.8, \n                                                  colsample_bytree=0.8,\n                                                  objective= 'binary:logistic', \n                                                  nthread=-1, \n                                                  scale_pos_weight=1, seed=2019),\nparam_grid = param_test1, scoring='accuracy',n_jobs=-1,iid=False, cv=5, verbose=10)\ngsearch1.fit(train[predictors],train[target])\ngsearch1.cv_results_, gsearch1.best_params_, gsearch1.best_score_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a08e83bb577730317f8392e7d362123940b5fdd8"},"cell_type":"markdown","source":"## 3-3. Gamma를 튜닝한다."},{"metadata":{"trusted":true,"_uuid":"c117e83d624842755aa6c203404517abbaca14bb"},"cell_type":"code","source":"param_test2 = {\n 'gamma':[i/10.0 for i in range(0,5)]\n}\ngsearch2 = GridSearchCV(estimator = XGBClassifier(learning_rate =0.1, \n                                                  n_estimators=1000, \n                                                  max_depth=3,\n                                                  min_child_weight=5, \n                                                  gamma=0, \n                                                  subsample=0.8, \n                                                  colsample_bytree=0.8,\n                                                  objective= 'binary:logistic', \n                                                  thread=-1, \n                                                  scale_pos_weight=1,\n                                                  seed=2019), \n                        param_grid = param_test2, scoring='accuracy', n_jobs=-1, iid=False, cv=5)\ngsearch2.fit(train[predictors],train[target])\ngsearch2.cv_results_, gsearch2.best_params_, gsearch2.best_score_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ca5d154c955108df4d85fa6ce93a92988398e71"},"cell_type":"markdown","source":"## 3-4. subsample and colsample_bytree를 튜닝한다.\n"},{"metadata":{"trusted":true,"_uuid":"cacb4eb6e8151f9d0afc431d64562545d46decf4"},"cell_type":"code","source":"param_test3 = {\n 'subsample':[i/10.0 for i in range(6,10)],\n 'colsample_bytree':[i/10.0 for i in range(6,10)]\n}\ngsearch3 = GridSearchCV(estimator = XGBClassifier(learning_rate =0.1, \n                                                  n_estimators=1000, \n                                                  max_depth=3,\n                                                  min_child_weight=5, \n                                                  gamma=0, \n                                                  subsample=0.8, \n                                                  colsample_bytree=0.8,\n                                                  objective= 'binary:logistic', \n                                                  thread=-1, \n                                                  scale_pos_weight=1,\n                                                  seed=2019), \n                        param_grid = param_test3, scoring='accuracy', n_jobs=-1, iid=False, cv=5, verbose=10)\ngsearch3.fit(train[predictors],train[target])\ngsearch3.cv_results_, gsearch3.best_params_, gsearch3.best_score_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"943a8f47bd2f1ad9a8165a5daa3326ebca468ca2"},"cell_type":"markdown","source":"## 3-4-2. subsample 추가 튜닝하기"},{"metadata":{"trusted":true,"_uuid":"7b0aa515a35950a4e157b035002b31e68cb0847b"},"cell_type":"code","source":"param_test4 = {\n 'subsample':[i/100.0 for i in range(40,80)],\n}\ngsearch4 = GridSearchCV(estimator = XGBClassifier(learning_rate =0.1, \n                                                  n_estimators=1000, \n                                                  max_depth=3,\n                                                  min_child_weight=5, \n                                                  gamma=0, \n                                                  subsample=0.6, \n                                                  colsample_bytree=0.8,\n                                                  objective= 'binary:logistic', \n                                                  thread=-1, \n                                                  scale_pos_weight=1,\n                                                  seed=2019), \n                        param_grid = param_test4, scoring='accuracy', n_jobs=-1, iid=False, cv=5, verbose=10)\ngsearch4.fit(train[predictors],train[target])\ngsearch4.cv_results_, gsearch4.best_params_, gsearch4.best_score_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2ba6dea09ce95f0dd43a234f50c3dd89b84deb5"},"cell_type":"markdown","source":"## 3-5. Regularization Parameter 튜닝"},{"metadata":{"trusted":true,"_uuid":"cedc1cb3ebde74c4cc9c094299ff74cc4b5e556e"},"cell_type":"code","source":"param_test5 = {\n 'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]\n}\ngsearch5 = GridSearchCV(estimator = XGBClassifier(learning_rate =0.1, \n                                                  n_estimators=1000, \n                                                  max_depth=3,\n                                                  min_child_weight=5, \n                                                  gamma=0, \n                                                  subsample=0.67, \n                                                  colsample_bytree=0.8,\n                                                  objective= 'binary:logistic', \n                                                  thread=-1, \n                                                  scale_pos_weight=1,\n                                                  seed=2019), \n                        param_grid = param_test5, scoring='accuracy', n_jobs=-1, iid=False, cv=5, verbose=10)\ngsearch5.fit(train[predictors],train[target])\ngsearch5.cv_results_, gsearch5.best_params_, gsearch5.best_score_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e3fbb8ec8e7555886583aca88d536cee71f6101"},"cell_type":"markdown","source":"## 3-6. Learning Rate 감소"},{"metadata":{"trusted":true,"_uuid":"8b2f67ec6d13c13f1a8de80599e6c5848a509a8f"},"cell_type":"code","source":"predictors = [x for x in train.columns if x not in [target, IDcol]]\nxgb1 = XGBClassifier(\n    learning_rate =0.01,\n    n_estimators=5000,\n    max_depth=3,\n    min_child_weight=5,\n    gamma=0,\n    reg_alpha=1e-05,\n    subsample=0.67,\n    colsample_bytree=0.8,\n    objective= 'binary:logistic',\n    nthread=-1,\n    scale_pos_weight=1,\n    seed=2019\n)\nmodelfit(xgb1, train, predictors)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"487b8b86ecd400f41cd3e1e5c9629bde1662516b"},"cell_type":"markdown","source":"# 4. seed별 앙상블 후 결과 제출"},{"metadata":{"trusted":true,"_uuid":"ea8d1997b41cf818fbf4abf688391cc9a79e1836"},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aa1e5e1594b73f5ccd797a345adb6ed3fa95ee63"},"cell_type":"code","source":"seeds = [2015, 2016, 2017, 2018, 2019]\npredictors = [x for x in train.columns if x not in [target, IDcol]]\n\nfor seed in seeds:\n    xgb1 = XGBClassifier(\n        learning_rate =0.01,\n        n_estimators=5000,\n        max_depth=3,\n        min_child_weight=5,\n        gamma=0,\n        reg_alpha=1e-05,\n        subsample=0.67,\n        colsample_bytree=0.8,\n        objective= 'binary:logistic',\n        nthread=-1,\n        scale_pos_weight=1,\n        seed=seed\n    )\n    modelfit(xgb1, train, predictors)\n    sample_submission['Survived'] += xgb1.predict(test[test.columns[2:]])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f06477c5beec7f5f3288e45da45cbd805b235261"},"cell_type":"markdown","source":"## 5. 결과 제출"},{"metadata":{"trusted":true,"_uuid":"9fdba2d98bd52bbb07fc767fc238ff0b33937ab6"},"cell_type":"code","source":"sample_submission['Survived'] = sample_submission['Survived'] > 2.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0463ba6bdcf4c3b65bfb479d8ab3739283b5d8ea"},"cell_type":"code","source":"sample_submission['Survived'] = sample_submission['Survived'].apply(lambda x: int(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd13411fc5534029dc26eaa8faa7b2ac264d921c"},"cell_type":"code","source":"sample_submission.to_csv('./my_third_submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"69ba24a843e38dd288568883ee35c7f3e3d5e27b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}