{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Catboost Stacking Solution\n\nDuring this competition I tried out stacking based on Catboost, Lightgbm and Xgboost. Catboost gave the best result of 0.9406 with few input predictions, but I did not get far with creating good input predictions and performing optimizations. In this notebook I want do demonstrate the stacking process with Catboost and show that this can lead to good results.\n\nIn notebook https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private Chris Deotte gives a sample ensemble based on forward selection achieving 0.9420 on private LB. Here I use the same input OOF and submission files, but use Catboost Classifier for stacking the predictions. In addition to the input predictions I add a single feature give by Giba in https://www.kaggle.com/titericz/simple-baseline. \n\nThe predictions on the different folds sometimes do to compare well for Catboost with auc dropping for the combined OOF predictions. Therefore I use the mean auc of the folds for hyperparameter selection. The hyperparameters are taken directly from above mentioned prediction I used for the competition.\n\nThe resulting predictions achieve 0.9432 on private LB.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np, os\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\nfrom catboost import CatBoostClassifier","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = '../input/melanoma-oof-and-sub/'\n\ncolumns_map = {\n    'age_approx': 'age',\n    'anatom_site_general_challenge': 'location',\n}\n\ndef get_train():    \n    df = pd.read_csv('../input/siim-isic-melanoma-classification/train.csv')\n    df['sex'].fillna('unknown', inplace=True)\n    df = df.rename(columns=columns_map)\n    df['location'].fillna('unknown', inplace=True)\n    \n    filenames = np.sort([f for f in os.listdir(PATH) if 'oof' in f])\n\n    for k, filename in enumerate(filenames):\n        df_preds = pd.read_csv(f'{PATH}{filename}')        \n        df = df.merge(df_preds[['image_name', 'pred']].rename(columns={'pred': f'p_{k}'}), how='inner', on=\"image_name\")\n        if k == 0:\n            df['fold'] = df_preds['fold']    \n    \n    return df\n\ndef get_test():\n    df = pd.read_csv('../input/siim-isic-melanoma-classification/test.csv')\n    df['sex'].fillna('unknown', inplace=True)\n    df = df.rename(columns=columns_map)\n    df['location'].fillna('unknown', inplace=True)\n    filenames = np.sort([f for f in os.listdir(PATH) if 'sub' in f])\n\n    for k, filename in enumerate(filenames):\n        df_preds = pd.read_csv(f'{PATH}{filename}')\n        df = df.merge(df_preds[['image_name', 'target']].rename(columns={'target': f'p_{k}'}), how='inner', on=\"image_name\")\n    \n    return df\n\ndef add_features(df_train, df_test):\n    df_train = pd.concat([df_train, pd.get_dummies(df_train['sex'], prefix='sex')], axis=1)\n    df_test = pd.concat([df_test, pd.get_dummies(df_test['sex'], prefix='sex')], axis=1)\n    \n    df_train = pd.concat([df_train, pd.get_dummies(df_train['location'], prefix='anatom')], axis=1)\n    df_test = pd.concat([df_test, pd.get_dummies(df_test['location'], prefix='anatom')], axis=1)\n    \n    L = 15\n    features = ['sex','age', 'location']\n\n    M = df_train.target.mean()\n    te = df_train.groupby(features)['target'].agg(['mean','count']).reset_index()\n    te['ll'] = ((te['mean']*te['count'])+(M*L))/(te['count']+L)\n    del te['mean'], te['count']\n\n    df_train = df_train.merge(te, on=features, how='left')\n    df_train['ll'] = df_train['ll'].fillna(M)\n\n    df_test = df_test.merge(te, on=features, how='left')\n    df_test['ll'] = df_test['ll'].fillna(M)  \n    \n    return df_train, df_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(df, params, features, nfolds=5, verbose_eval=10):\n\n    folds = range(nfolds)\n    target = 'target'\n    \n    df_oof = df[['image_name', 'target', 'fold']].copy()\n    y = df[target].values\n    oof_preds = np.zeros((df_oof.shape[0], 1))\n\n    scores = []\n\n    for fold in folds:\n        print(f'==========> Training fold {fold} <==========')\n        df_train = df[df.fold != fold]\n        df_valid = df[df.fold == fold]\n        valid_idx = df_valid.index\n\n        X_train, y_train = df_train[features], df_train[target]\n        X_valid, y_valid = df_valid[features], df_valid[target]\n\n        run_params = params.copy()\n        model = CatBoostClassifier(**run_params)\n        model.fit(X_train, y_train,\n                  eval_set=(X_valid, y_valid),\n                  use_best_model=True,                  \n                  early_stopping_rounds=50,\n                  verbose_eval=10)\n\n        y_preds = model.predict_proba(X_valid)[:,1]\n#         y_preds = (y_preds - y_preds.min())/(y_preds.max() - y_preds.min())\n        score_fold = roc_auc_score(y_valid, y_preds)\n        print(f'auc roc score fold {fold} auc: {score_fold}') \n        scores.append(score_fold)\n        oof_preds[valid_idx] = y_preds.reshape(-1,1)\n        df_oof.loc[df_oof.fold == fold, \"pred\"] = y_preds.reshape(-1,1)\n\n        model.save_model(f'model_{fold}')\n\n    score = roc_auc_score(y, df_oof['pred'])\n    print(f'auc roc score: {score}')\n    print(f'Mean auc roc score per fold: {sum(scores)/len(scores):0.5f}')\n\n    return df_oof","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict(df, modelpaths, features):\n    print(f'======================> Start Predictions <======================')\n    df_preds = df[['image_name']].copy()\n    df_preds['target'] = 0.0\n\n    y_preds = np.zeros((df.shape[0], 1))\n    X = df[features].values\n\n    n = len(modelpaths)\n\n    for k, modelpath in enumerate(modelpaths):\n        print(f'Predicting with model {k}')\n        model = CatBoostClassifier() \n        model.load_model(modelpath)\n        y_preds += model.predict_proba(X)[:,1].reshape(-1,1)/n\n        \n    df_preds['target'] = y_preds\n\n    return df_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_submission(df_preds):\n    submission = pd.read_csv(\"../input/siim-isic-melanoma-classification/sample_submission.csv\")\n    submission.target = df_preds.target.values.reshape(-1)\n    submission.to_csv(f'submissions.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = get_train()\ndf_test = get_test()\ndf_train, df_test = add_features(df_train, df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [\n    'll',\n    'p_1',\n    'p_3',\n    'p_8',\n    'p_10',\n    'p_12',\n    'p_21',\n    'p_26',\n    'p_37',\n]\n\nparams = {\n    'loss_function': 'CrossEntropy',\n    'eval_metric': 'AUC',\n    'bootstrap_type': 'Bayesian',\n    'bagging_temperature': 0.5,\n    'grow_policy': 'SymmetricTree',\n    'depth': 3,\n    'learning_rate': 0.03,\n    'iterations': 2000,\n}\n\ndf_oof = train(df_train, params, features, nfolds=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"modelpaths = [f'model_{k}' for k in range(5)]\ndf_preds = predict(df_test, modelpaths, features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"create_submission(df_preds)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}