Basic Understanding of Ensemble Models
\n### A Work In Progress by Ed Welch\n# Topics: \n> - [Introduction to Ensembling](#1)\n> - [Understanding Error](#10)\n> - --- [Bias](#2) -- \n> - ---[Variance](#3) -- \n> - [Bias/Variance Tradeoff](#20)\n> - [Why Ensembles?](#4) \n> - [Types of ensembling](#5)\n> - --- [Stacking](#6)\n> - ------- [Choosing Models for your Stack](#30)\n> - --- [Bagging](#7) --\n> - --- [Boosting](#8) --","metadata":{}},{"cell_type":"markdown","source":"### System Updates","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade scikit-learn\n!pip install mlxtend ","metadata":{"execution":{"iopub.status.busy":"2022-07-04T15:24:49.131415Z","iopub.execute_input":"2022-07-04T15:24:49.131987Z","iopub.status.idle":"2022-07-04T15:25:16.874444Z","shell.execute_reply.started":"2022-07-04T15:24:49.131951Z","shell.execute_reply":"2022-07-04T15:25:16.873347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.ensemble import BaggingClassifier\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.ensemble import StackingClassifier\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.ensemble import ExtraTreesClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom xgboost import XGBClassifier\nimport plotly.express as px\nfrom scipy import stats\nimport re\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n \nimport warnings\nwarnings.filterwarnings('ignore')\nlabel_encoder = LabelEncoder()\n\ntrain = pd.read_csv(\"../input/tabular-playground-series-feb-2022/train.csv\")\ntest = pd.read_csv(\"../input/tabular-playground-series-feb-2022/test.csv\")\nsubmission = pd.read_csv(\"../input/tabular-playground-series-feb-2022/sample_submission.csv\")\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-07-04T15:25:16.878088Z","iopub.execute_input":"2022-07-04T15:25:16.878468Z","iopub.status.idle":"2022-07-04T15:25:43.022205Z","shell.execute_reply.started":"2022-07-04T15:25:16.878431Z","shell.execute_reply":"2022-07-04T15:25:43.021292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INCREASE THE SPEED OF THE MODEL BY REDUCING THE DATA, BUT REDUCED DATA = LOWER MODEL ACCURACY\ntrain = train.sample(frac=1).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-04T15:25:43.023639Z","iopub.execute_input":"2022-07-04T15:25:43.02408Z","iopub.status.idle":"2022-07-04T15:25:43.707103Z","shell.execute_reply.started":"2022-07-04T15:25:43.024032Z","shell.execute_reply":"2022-07-04T15:25:43.706467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Introduction to Ensemble Models
\n\n> Ensembling Defined: \"In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.\" - [Wikipedia](https://en.wikipedia.org/wiki/Ensemble_learning)\n\n# Error and Why Ensembling Works