{
  "id": 276352,
  "title": "How to ensemble of Out-of-folds model (same configuratoin)? ",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/276352",
  "author_name": "",
  "post_date": "2021-10-04T11:56:12.605008200Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>After training with same configuration, what are the strategies to ensemble the models? Any suggestion? </p>",
  "messages": [
    {
      "id": "1533836",
      "postDate": "10/04/2021 11:56:12",
      "content": "<p>After training with same configuration, what are the strategies to ensemble the models? Any suggestion? </p>",
      "rawMarkdown": "After training with same configuration, what are the strategies to ensemble the models? Any suggestion?",
      "votes": null
    },
    {
      "id": "1534069",
      "postDate": "10/04/2021 15:19:14",
      "content": "<p>Average the probability of each model.</p>",
      "rawMarkdown": "Average the probability of each model.",
      "votes": null
    },
    {
      "id": "1534475",
      "postDate": "10/05/2021 00:29:47",
      "content": "<p>As you're making the <strong>same configuration</strong> on a different fold, you shouldn't do just a simple mean of the predictions. Instead, you can adopt some mechanism to search linear weights to weighting each prediction.  One of two such mechanisms are </p>\n<pre><code>- ScipyOptimize (L-BFGS-B)\n- BayesianOptimization\n</code></pre>\n<p>In <code>scipy</code>, with <code>L-BFGS-B</code> method:</p>\n<pre><code>from scipy.optimize import minimize\n\n# metric function to minimize\ndef roc_min_func(weights):\n    final_prediction = 0\n    for weight, prediction in zip(weights, blend_train):\n        final_prediction += weight * prediction\n    return roc_auc_score(np.array(oof_one.target), final_prediction)\n\nres = minimize(roc_min_func,\n                   starting_values,\n                   method='L-BFGS-B',\n                   bounds=bounds,\n                   options={'disp': False,\n                            'maxiter': 100000}) \n</code></pre>\n<p>And, in <code>bayseian</code>, you would do: </p>\n<pre><code>from bayes_opt import BayesianOptimization\n\noptimizer = BayesianOptimization(\n        f=q,\n        pbounds=pbounds,\n        random_state=42,\n    )\n\noptimizer.maximize(\n        init_points=init_points,\n        n_iter=n_iter,\n    )\n</code></pre>\n<p>Check out this notebook for end-to-end code examples. Hope it helps. </p>\n<ul>\n<li><a href=\"https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble/notebook\" target=\"_blank\">Optimizing Metrics: Out-of-Fold Weights Ensemble</a></li>\n</ul>",
      "rawMarkdown": "As you're making the **same configuration** on a different fold, you shouldn't do just a simple mean of the predictions. Instead, you can adopt some mechanism to search linear weights to weighting each prediction.  One of two such mechanisms are \n\n```\n- ScipyOptimize (L-BFGS-B)\n- BayesianOptimization\n```\n\nIn `scipy`, with `L-BFGS-B` method:\n\n```\nfrom scipy.optimize import minimize\n\n# metric function to minimize\ndef roc_min_func(weights):\n    final_prediction = 0\n    for weight, prediction in zip(weights, blend_train):\n        final_prediction += weight * prediction\n    return roc_auc_score(np.array(oof_one.target), final_prediction)\n\nres = minimize(roc_min_func,\n                   starting_values,\n                   method='L-BFGS-B',\n                   bounds=bounds,\n                   options={'disp': False,\n                            'maxiter': 100000}) \n```\n\nAnd, in `bayseian`, you would do: \n\n```\nfrom bayes_opt import BayesianOptimization\n\noptimizer = BayesianOptimization(\n        f=q,\n        pbounds=pbounds,\n        random_state=42,\n    )\n\noptimizer.maximize(\n        init_points=init_points,\n        n_iter=n_iter,\n    )\n```\n\nCheck out this notebook for end-to-end code examples. Hope it helps. \n\n- [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble/notebook)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1534069,
      "author_name": "hyeonhoonlee",
      "author_url": "",
      "post_date": "10/04/2021 15:19:14",
      "content": "<p>Average the probability of each model.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1534475,
      "author_name": "ipythonx",
      "author_url": "",
      "post_date": "10/05/2021 00:29:47",
      "content": "<p>As you're making the <strong>same configuration</strong> on a different fold, you shouldn't do just a simple mean of the predictions. Instead, you can adopt some mechanism to search linear weights to weighting each prediction.  One of two such mechanisms are </p>\n<pre><code>- ScipyOptimize (L-BFGS-B)\n- BayesianOptimization\n</code></pre>\n<p>In <code>scipy</code>, with <code>L-BFGS-B</code> method:</p>\n<pre><code>from scipy.optimize import minimize\n\n# metric function to minimize\ndef roc_min_func(weights):\n    final_prediction = 0\n    for weight, prediction in zip(weights, blend_train):\n        final_prediction += weight * prediction\n    return roc_auc_score(np.array(oof_one.target), final_prediction)\n\nres = minimize(roc_min_func,\n                   starting_values,\n                   method='L-BFGS-B',\n                   bounds=bounds,\n                   options={'disp': False,\n                            'maxiter': 100000}) \n</code></pre>\n<p>And, in <code>bayseian</code>, you would do: </p>\n<pre><code>from bayes_opt import BayesianOptimization\n\noptimizer = BayesianOptimization(\n        f=q,\n        pbounds=pbounds,\n        random_state=42,\n    )\n\noptimizer.maximize(\n        init_points=init_points,\n        n_iter=n_iter,\n    )\n</code></pre>\n<p>Check out this notebook for end-to-end code examples. Hope it helps. </p>\n<ul>\n<li><a href=\"https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble/notebook\" target=\"_blank\">Optimizing Metrics: Out-of-Fold Weights Ensemble</a></li>\n</ul>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1533836": "After training with same configuration, what are the strategies to ensemble the models? Any suggestion?",
    "1534069": "Average the probability of each model.",
    "1534475": "As you're making the **same configuration** on a different fold, you shouldn't do just a simple mean of the predictions. Instead, you can adopt some mechanism to search linear weights to weighting each prediction.  One of two such mechanisms are \n\n```\n- ScipyOptimize (L-BFGS-B)\n- BayesianOptimization\n```\n\nIn `scipy`, with `L-BFGS-B` method:\n\n```\nfrom scipy.optimize import minimize\n\n# metric function to minimize\ndef roc_min_func(weights):\n    final_prediction = 0\n    for weight, prediction in zip(weights, blend_train):\n        final_prediction += weight * prediction\n    return roc_auc_score(np.array(oof_one.target), final_prediction)\n\nres = minimize(roc_min_func,\n                   starting_values,\n                   method='L-BFGS-B',\n                   bounds=bounds,\n                   options={'disp': False,\n                            'maxiter': 100000}) \n```\n\nAnd, in `bayseian`, you would do: \n\n```\nfrom bayes_opt import BayesianOptimization\n\noptimizer = BayesianOptimization(\n        f=q,\n        pbounds=pbounds,\n        random_state=42,\n    )\n\noptimizer.maximize(\n        init_points=init_points,\n        n_iter=n_iter,\n    )\n```\n\nCheck out this notebook for end-to-end code examples. Hope it helps. \n\n- [Optimizing Metrics: Out-of-Fold Weights Ensemble](https://www.kaggle.com/ipythonx/optimizing-metrics-out-of-fold-weights-ensemble/notebook)"
  },
  "source": "meta"
}