{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Update**\n\nWe've included `Bayesian Optimization` from [this notebook](https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification). Thanks to [steubk](https://www.kaggle.com/steubk). Aim was to compare with `scipy.optimize` function. \n\n---\n\nIn this notebook, we attempt to find the best weight vector for ensembling via trying to maximize **OOF CV** scores. The piece of code actually came from [this work](https://www.kaggle.com/tilii7/cross-validation-weighted-linear-blending-errors). \n\n\n**Disclaimer**\n\nAll the input scripts are designed for demonstratio purpose and also the procedure tha we've presented here. Please, use it to your own risk. Also note, in order to use it, you Have-To-Have **N** times `oof.csv` and `test.csv` that trained one the **same validation fold**. Thanks.\n\n---\n\nSet Up:\n\n```\nModel: E6\nSeed : 42\nExp  : 5 Fold Training, Total 4 experiment [same fold trainig]\n```","metadata":{}},{"cell_type":"code","source":"import os\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom bayes_opt import BayesianOptimization\nfrom scipy.optimize import minimize\nfrom sklearn.metrics import roc_auc_score\n\ndir = \"../input/e6-oof-prediction//\"\nnp.random.seed(42)\n\n# exp 1 ___________\noof_one = pd.read_csv(dir + \"oof_e6_exp1_seed_42.csv\")\ntest_one = pd.read_csv(dir + \"s_e6_exp1_seed_42.csv\")\noof_one = oof_one.sort_values(by=[\"image_name\"], ascending=True).reset_index(drop=True)\ntest_one = test_one.sort_values(by=[\"image_name\"], ascending=True).reset_index(\n    drop=True\n)\n\n# exp 2 ___________\noof_two = pd.read_csv(dir + \"oof_e6_exp2_seed_42.csv\")\ntest_two = pd.read_csv(dir + \"s_e6_exp2_seed_42.csv\")\noof_two = oof_two.sort_values(by=[\"image_name\"], ascending=True).reset_index(drop=True)\ntest_two = test_two.sort_values(by=[\"image_name\"], ascending=True).reset_index(\n    drop=True\n)\n\n# exp 3 ___________\noof_three = pd.read_csv(dir + \"oof_e6_exp3_seed_42.csv\")\ntest_three = pd.read_csv(dir + \"s_e6_exp3_seed_42.csv\")\noof_three = oof_three.sort_values(by=[\"image_name\"], ascending=True).reset_index(\n    drop=True\n)\ntest_three = test_three.sort_values(by=[\"image_name\"], ascending=True).reset_index(\n    drop=True\n)\n\n# exp 4 ___________\noof_four = pd.read_csv(dir + \"oof_e6_exp4_seed_42.csv\")\ntest_four = pd.read_csv(dir + \"s_e6_exp4_seed_42.csv\")\noof_four = oof_four.sort_values(by=[\"image_name\"], ascending=True).reset_index(\n    drop=True\n)\ntest_four = test_four.sort_values(by=[\"image_name\"], ascending=True).reset_index(\n    drop=True\n)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:51.770543Z","iopub.execute_input":"2022-07-02T07:24:51.771139Z","iopub.status.idle":"2022-07-02T07:24:52.044887Z","shell.execute_reply.started":"2022-07-02T07:24:51.771104Z","shell.execute_reply":"2022-07-02T07:24:52.043968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_one.fold.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:52.047386Z","iopub.execute_input":"2022-07-02T07:24:52.047803Z","iopub.status.idle":"2022-07-02T07:24:52.058479Z","shell.execute_reply.started":"2022-07-02T07:24:52.047762Z","shell.execute_reply":"2022-07-02T07:24:52.057498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_one.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:52.059737Z","iopub.execute_input":"2022-07-02T07:24:52.059987Z","iopub.status.idle":"2022-07-02T07:24:52.075503Z","shell.execute_reply.started":"2022-07-02T07:24:52.059962Z","shell.execute_reply":"2022-07-02T07:24:52.074790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_one.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:52.076642Z","iopub.execute_input":"2022-07-02T07:24:52.077144Z","iopub.status.idle":"2022-07-02T07:24:52.090550Z","shell.execute_reply.started":"2022-07-02T07:24:52.077102Z","shell.execute_reply":"2022-07-02T07:24:52.089778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Scipy Optimizer\n\nLet's try first optmization function from `Scipy`, usng `L-BFGS-B` method. We've tried other methods, but this one gave best.","metadata":{}},{"cell_type":"code","source":"blend_train = []\nblend_test = []\n\n# out of fold prediction\nblend_train.append(oof_one.pred)\nblend_train.append(oof_two.pred)\nblend_train.append(oof_three.pred)\nblend_train.append(oof_four.pred)\nblend_train = np.array(blend_train)\n\n# submission scores\nblend_test.append(test_one.target)\nblend_test.append(test_two.target)\nblend_test.append(test_three.target)\nblend_test.append(test_four.target)\nblend_test = np.array(blend_test)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:52.092406Z","iopub.execute_input":"2022-07-02T07:24:52.092922Z","iopub.status.idle":"2022-07-02T07:24:52.146551Z","shell.execute_reply.started":"2022-07-02T07:24:52.092892Z","shell.execute_reply":"2022-07-02T07:24:52.145822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"a = 0.12345\nprint(\"{0:.2f}\".format(a))","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:52.148041Z","iopub.execute_input":"2022-07-02T07:24:52.148422Z","iopub.status.idle":"2022-07-02T07:24:52.154592Z","shell.execute_reply.started":"2022-07-02T07:24:52.148393Z","shell.execute_reply":"2022-07-02T07:24:52.153559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def roc_min_func(weights):\n    final_prediction = 0\n    for weight, prediction in zip(weights, blend_train):\n        final_prediction += weight * prediction\n    return roc_auc_score(np.array(oof_one.target), final_prediction)\n\n\nprint(\"\\n Finding Blending Weights ...\")\nres_list = []\nweights_list = []\n\nfor k in range(100):\n    starting_values = np.random.uniform(size=len(blend_train))\n    bounds = [(0, 1)] * len(blend_train)\n\n    res = minimize(\n        roc_min_func,\n        starting_values,\n        method=\"L-BFGS-B\",\n        bounds=bounds,\n        options={\"disp\": False, \"maxiter\": 100000},\n    )\n\n    res_list.append(res[\"fun\"])\n    weights_list.append(res[\"x\"])\n\n    \n    res_fun = format(res[\"fun\"], '.7f')\n    items = [format(item, '.7f') for item in res[\"x\"]]\n    \n    print(\n        \"{iter}\\tScore: {score}\\tWeights: {weights}\".format(\n            iter=(k + 1),\n            score=res_fun,\n            weights=\"\\t\".join([str(item) for item in items]),\n        )\n    )\n\nbestSC = np.max(res_list)\nbestWght = weights_list[np.argmax(res_list)]\nweights = bestWght\nblend_score = round(bestSC, 6)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:52.155782Z","iopub.execute_input":"2022-07-02T07:24:52.156147Z","iopub.status.idle":"2022-07-02T07:24:57.137481Z","shell.execute_reply.started":"2022-07-02T07:24:52.156115Z","shell.execute_reply":"2022-07-02T07:24:57.136251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\n Ensemble Score: {best_score}'.format(best_score=bestSC))\nprint('\\n Best Weights: {weights}'.format(weights=bestWght))\n\ntrain_prices = np.zeros(len(blend_train[0]))\ntest_prices  = np.zeros(len(blend_test[0]))\n\nprint('\\n Your final model:')\nfor k in range(len(blend_test)):\n    print(' %.6f * model-%d' % (weights[k], (k + 1)))\n    test_prices += blend_test[k] * weights[k]\n\nfor k in range(len(blend_train)):\n    train_prices += blend_train[k] * weights[k]","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:57.140516Z","iopub.execute_input":"2022-07-02T07:24:57.140946Z","iopub.status.idle":"2022-07-02T07:24:57.150921Z","shell.execute_reply.started":"2022-07-02T07:24:57.140915Z","shell.execute_reply":"2022-07-02T07:24:57.150036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_one.target = (\n    test_one.target.values * bestWght[0]\n    + test_two.target.values * bestWght[1]\n    + test_three.target.values * bestWght[2]\n    + test_four.target.values * bestWght[3]\n) / max(bestWght)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:57.153645Z","iopub.execute_input":"2022-07-02T07:24:57.154042Z","iopub.status.idle":"2022-07-02T07:24:57.169523Z","shell.execute_reply.started":"2022-07-02T07:24:57.154008Z","shell.execute_reply":"2022-07-02T07:24:57.168637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(test_one.target,bins=100)\nplt.ylim((0,100))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:57.170892Z","iopub.execute_input":"2022-07-02T07:24:57.171175Z","iopub.status.idle":"2022-07-02T07:24:57.549882Z","shell.execute_reply.started":"2022-07-02T07:24:57.171146Z","shell.execute_reply":"2022-07-02T07:24:57.548781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Bayesian Optimization\n\nNext, Bayesian. ","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')\ntest = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/test.csv')\nsub = pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:57.551274Z","iopub.execute_input":"2022-07-02T07:24:57.551563Z","iopub.status.idle":"2022-07-02T07:24:57.626360Z","shell.execute_reply.started":"2022-07-02T07:24:57.551535Z","shell.execute_reply":"2022-07-02T07:24:57.625472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dirname = '/kaggle/input/e6-oof-prediction'\nprefix = ['exp1', 'exp2', 'exp3', 'exp4', ]\n\nfor model in range(4):\n    _oof = pd.read_csv(os.path.join(dirname, f\"oof_e6_{prefix[model]}_seed_42.csv\"))\n    score = roc_auc_score(_oof['target'], _oof['pred'])\n    print(f\"{prefix[model]}: OOF auc:{score:.4}\")\n\n    _oof = _oof.rename(columns={\"pred\":prefix[model]}).drop([\"target\"],axis=1)\n    \n    if \"fold\" in _oof.columns:\n        _oof = _oof.drop([\"fold\"],axis=1)\n\n    train = train.merge(_oof, on=\"image_name\")   \n\n    _sub = pd.read_csv(os.path.join(dirname, f\"s_e6_{prefix[model]}_seed_42.csv\"))\n    _sub.columns = [\"image_name\", prefix[model]]    \n    test = test.merge(_sub, on=\"image_name\")  ","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:57.627668Z","iopub.execute_input":"2022-07-02T07:24:57.627974Z","iopub.status.idle":"2022-07-02T07:24:57.972953Z","shell.execute_reply.started":"2022-07-02T07:24:57.627946Z","shell.execute_reply":"2022-07-02T07:24:57.971906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dim_optimizer (df_oof, features, init_points = 20, n_iter = 100 ):\n    pbounds = {'c0': (0.0, 1.0), \n               'c1': (0.0, 1.0), \n               'c2': (0.0, 1.0), \n               'c3': (0.0, 1.0)}\n    \n    features = features\n\n    def dim_opt (df_oof, c0,c1,c2,c3):\n\n        x = (c0*df_oof[features[0]] + \n             c1*df_oof[features[1]] + \n             c2*df_oof[features[2]] + \n             c3*df_oof[features[3]])\n        \n        return roc_auc_score(df_oof['target'], x)\n\n\n\n    def q (c0,c1,c2,c3):\n        return dim_opt  ( df_oof, c0,c1,c2,c3)\n\n    optimizer = BayesianOptimization(\n        f=q,\n        pbounds=pbounds,\n        random_state=42,\n    )\n\n\n    optimizer.maximize(\n        init_points=init_points,\n        n_iter=n_iter,\n    )\n\n    c0 = optimizer.max[\"params\"][\"c0\"]\n    c1 = optimizer.max[\"params\"][\"c1\"]\n    c2 = optimizer.max[\"params\"][\"c2\"]\n    c3 = optimizer.max[\"params\"][\"c3\"]\n    t  = optimizer.max[\"target\"]\n    \n    print ( f'bo auc:{t}, c0:{c0}, c1:{c1}, c2:{c2}, c3:{c3}' )\n    \n    return c0, c1, c2, c3\n\n\nc0, c1, c2, c3 = dim_optimizer (train, prefix, \n                                init_points = 40, \n                                n_iter = 40  )\n\nprint(' ')\nprint (prefix[0],c0)\nprint (prefix[1],c1)\nprint (prefix[2],c2)\nprint (prefix[3],c3)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T07:24:57.974188Z","iopub.execute_input":"2022-07-02T07:24:57.974469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def bo_pred(df):\n    x = (\n        c0 * df[prefix[0]]\n        + c1 * df[prefix[1]]\n        + c2 * df[prefix[2]]\n        + c3 * df[prefix[3]]\n    )\n\n    return x\n\n\ntrain[\"pred\"] = bo_pred(train)\nscore = roc_auc_score(train[\"target\"], train[\"pred\"])\nprint(f\"auc bo:{score}\")\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(train.pred,bins=100)\nplt.ylim((0,100))\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compare L-BFGS-B vs Bayesian Operations\n\n\n|  Method | CV  |  \n|---|---|\n| L-BFGS-B  | 0.9316942291023538  |   \n| Bayesian-Op  | 0.9316718240316467  |","metadata":{}}]}