{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "3b3c1c94-6864-4979-8e9e-8323063514b3" }, "source": [ "# Learning Curves\n", "\n", "### Learning Curves help diagnose Bias and Variance" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e0277791-3d13-8c1e-9904-a642bffe1fd1" }, "source": [ "![model complexity](http://scott.fortmann-roe.com/docs/docs/BiasVariance/biasvariance.png)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b8a873aa-bd40-3e01-1d40-c3cd47281d2e" }, "source": [ "A learning curve shows the validation and training score of an estimator for varying numbers of training samples. It is a tool to find out how much we benefit from adding more training data and whether the estimator suffers more from a variance error or a bias error. \n", "\n", "If the training score is much greater than the validation score for the maximum number of training samples, adding more training samples will most likely increase generalization. \n", "\n", "Variance is the amount that the estimate of the target function will change if different training data is used. \n", "\n", " * Examples of low-variance/high-bias algorithms: Linear Regression, Linear Discriminant Analysis and Logistic Regression.\n", "\n", " * Examples of high-variance/low-bias algorithms: Decision Trees (especially if not pruned), k-Nearest Neighbors and Support Vector Machines.\n", "\n", "\n", "If both the validation score and the training score converge to a value that is too low with increasing size of the training set, we will not benefit much from more training data. We will probably have to use an estimator or a parametrization of the current estimator that can learn more complex concepts (i.e. has a lower bias). \n", "\n", "**High Bias** => Increase model complexity: add features \n", "**High Variance** => Reduce model complexity \n", "\n", "\n", "### Reduce Variance Without Increasing Bias \n", " * Averaging reduces variance: \n", " \n", "$Var(\\bar{X}) = \\frac{Var(X)}{N}$ \n", " \n", " * bagging \n", " * boosting \n", " * feature elimination \n", " * regularization \n", " * low-variance filter \n", " * PCA \n", " * RFE \n", " * ensembling \n", " \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ed3c974e-bfef-a072-6fef-d17b9fe21dd2" }, "outputs": [], "source": [ "import numpy as np\n", "\n", "def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None, scoring=None, obj_line=None,\n", " n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5)):\n", " \"\"\"\n", " Generate a simple plot of the test and training learning curve.\n", "\n", " Parameters\n", " ----------\n", " estimator : object type that implements the \"fit\" and \"predict\" methods\n", " An object of that type which is cloned for each validation.\n", "\n", " title : string\n", " Title for the chart.\n", "\n", " X : array-like, shape (n_samples, n_features)\n", " Training vector, where n_samples is the number of samples and\n", " n_features is the number of features.\n", "\n", " y : array-like, shape (n_samples) or (n_samples, n_features), optional\n", " Target relative to X for classification or regression;\n", " None for unsupervised learning.\n", "\n", " ylim : tuple, shape (ymin, ymax), optional\n", " Defines minimum and maximum yvalues plotted.\n", "\n", " cv : int, cross-validation generator or an iterable, optional\n", " Determines the cross-validation splitting strategy.\n", " Possible inputs for cv are:\n", " - None, to use the default 3-fold cross-validation,\n", " - integer, to specify the number of folds.\n", " - An object to be used as a cross-validation generator.\n", " - An iterable yielding train/test splits.\n", "\n", " For integer/None inputs, if ``y`` is binary or multiclass,\n", " :class:`StratifiedKFold` used. If the estimator is not a classifier\n", " or if ``y`` is neither binary nor multiclass, :class:`KFold` is used.\n", "\n", " Refer :ref:`User Guide ` for the various\n", " cross-validators that can be used here.\n", "\n", " scoring : string, callable or None, optional, default: None\n", " A string (see model evaluation documentation)\n", " or a scorer callable object / function with signature scorer(estimator, X, y)\n", " For Python 3.5 the documentation is here:\n", " http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter\n", " For example, Log Loss is specified as 'neg_log_loss'\n", "\n", " obj_line : numeric or None (default: None)\n", " draw a horizontal line\n", "\n", "\n", " n_jobs : integer, optional\n", " Number of jobs to run in parallel (default 1).\n", "\n", "\n", " Citation\n", " --------\n", " http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html\n", "\n", " Usage\n", " -----\n", " plot_learning_curve(estimator = best_estimator,\n", " title = best_estimator_title,\n", " X = X_train,\n", " y = y_train,\n", " ylim = (-1.1, 0.1), # neg_log_loss is negative\n", " cv = StatifiedCV, # CV generator\n", " scoring = scoring, # eg., 'neg_log_loss'\n", " obj_line = obj_line, # horizontal line\n", " n_jobs = n_jobs) # how many CPUs\n", "\n", " plt.show()\n", " \"\"\"\n", " from sklearn.model_selection import learning_curve\n", " import numpy as np\n", " from matplotlib import pyplot as plt\n", "\n", " plt.figure()\n", " plt.title(title)\n", " if ylim is not None:\n", " plt.ylim(*ylim)\n", " plt.xlabel(\"Training examples\")\n", " plt.ylabel(\"Score\")\n", " train_sizes, train_scores, test_scores = learning_curve(\n", " estimator, X, y, cv=cv, scoring=scoring, n_jobs=n_jobs, train_sizes=train_sizes)\n", " train_scores_mean = np.mean(train_scores, axis=1)\n", " train_scores_std = np.std(train_scores, axis=1)\n", " test_scores_mean = np.mean(test_scores, axis=1)\n", " test_scores_std = np.std(test_scores, axis=1)\n", " plt.grid()\n", "\n", " plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n", " train_scores_mean + train_scores_std, alpha=0.1,\n", " color=\"r\")\n", " plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n", " test_scores_mean + test_scores_std, alpha=0.1, color=\"g\")\n", " plt.plot(train_sizes, train_scores_mean, 'o-', color=\"r\",\n", " label=\"Training score\")\n", " plt.plot(train_sizes, test_scores_mean, 'o-', color=\"g\",\n", " label=\"Cross-validation score\")\n", "\n", " if obj_line:\n", " plt.axhline(y=obj_line, color='blue')\n", "\n", " plt.legend(loc=\"best\")\n", " return plt\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c741cc8e-883f-80ce-50a4-5fd8a48eaa6a" }, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn.naive_bayes import GaussianNB\n", "from sklearn.svm import SVC\n", "\n", "from sklearn.datasets import load_digits\n", "from sklearn.model_selection import ShuffleSplit\n", "from sklearn.model_selection import StratifiedKFold\n", "\n", "digits = load_digits()\n", "X, y = digits.data, digits.target\n", "\n", "estimator = GaussianNB()\n", "plot_learning_curve(estimator = estimator,\n", " title = \"Learning Curves (Naive Bayes)\",\n", " X = X,\n", " y = y,\n", " ylim = (0.5, 1.1),\n", " cv = StratifiedKFold(),\n", " scoring = 'accuracy', \n", " obj_line = 0.85, \n", " n_jobs = -1) \n", "plt.show()\n", "\n", "\n", "estimator = SVC(gamma=0.001)\n", "plot_learning_curve(estimator = estimator,\n", " title = \"Learning Curves (SVM, RBF kernel, $\\gamma=0.001$)\",\n", " X = X,\n", " y = y,\n", " ylim = (0.8, 1.1),\n", " cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0),\n", " scoring = 'accuracy', \n", " obj_line = 0.99, \n", " n_jobs = -1)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bd2e2099-1744-d1f7-34d0-7b3ebbabd7ee" }, "source": [ "![andrew ng](http://www.ultravioletanalytics.com/wp-content/uploads/2014/12/bias_variance_chart1.jpg) \n", "\n", "### c/o Andrew Ng" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "fcd153e0-585e-5bdc-904b-4dabd3f4ebbe" }, "outputs": [], "source": "" } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }