{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import make_scorer\nimport lightgbm as lgb\nimport types","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"We all love the power of learning libraries like Keras, XGBoost or LightGBM, and we all love SkLearn for offering a unified API for many different tools we use as well as augmenting them with a set of additional tools like metrics, CV, feature selection etc. However, sometimes your favourite machine learning library might not play nicely with SkLearn. LightGBM supports categorical variables, but SkLearn mostly does not. Many tree-based libraries can utilise missing values natively, but SkLearn whines about NaNs all the time. Keras may need some additional tweaking on that `fit()` call via parameters, but so many SkLearn interfaces restrict `fit()` to take only X and y... However, we are not helpless. While monkey patching is often frowned upon in more serious settings, this is where it can really help us. Here are some examples to make SkLearn work nicely with LightGBM, but the same principle applies to any library."},{"metadata":{"_uuid":"bdeb1af56a4f715459d6cf21e3c1a35690d6525e"},"cell_type":"markdown","source":"## 1. Stop SkLearn from complaining about NaN or Inf"},{"metadata":{"_uuid":"b756af5c3fdd54d9e7643a7cdaf89e2afaa1a55e"},"cell_type":"markdown","source":"Suppose you want to use the fancy automatic recursive feature elimination in SkLearn for your LightGBM classifier. So as usual, you will set up the code:\n```\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=1111)\nclf_params = {\n    ...\n    }\nclf = lgb.LGBMClassifier(**clf_params)\ncv = RFECV(clf, step=5, min_features_to_select=50, cv=folds, verbose=10,\n               scoring=make_scorer(wloss, greater_is_better=False, needs_proba=True))\ncv.fit(X, y)               \n```\nBut then you are immediately told you cannot do so because SkLearn checked your inputs and there are missing values or infinite values. Looking through the trace stack, you can pinpoint the issue to a call to `check_X_y()` or `check_array()`. Looking at the definition of these functions, you will notice that they have a parameter called `force_all_finite`. Here is the documentation:\n```\nforce_all_finite : boolean or 'allow-nan', (default=True)\n        Whether to raise an error on np.inf and np.nan in X. This parameter\n        does not influence whether y can have np.inf or np.nan values.\n        The possibilities are:\n\n        - True: Force all values of X to be finite.\n        - False: accept both np.inf and np.nan in X.\n        - 'allow-nan':  accept  only  np.nan  values in  X.  Values  cannot  be\n          infinite.\n```\nSo we can easily solve this problem with setting it to `False`, right? Unfortunately we don't call this function, but SkLearn does. We can alter the SkLearn library source code, but that usually is a bad idea. What we can do is to replace that one method or one module. In this case, we can simply make a copy of the whole `rfe` module and add `force_all_finite=True` to the `check_X_y()` and `check_array()` calls. The modified code this hidden below. It can be saved to a separate file for importing the necessary classes instead of from OG SkLearn. You may have to change all `..` in imports at the beginning of the file to `sklearn.`."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f0129c3955d540d80c08a1d843f9896c71485b3c"},"cell_type":"code","source":"import numpy as np\nfrom sklearn.utils import check_X_y, safe_sqr\nfrom sklearn.utils.metaestimators import if_delegate_has_method\nfrom sklearn.utils.metaestimators import _safe_split\nfrom sklearn.utils.validation import check_is_fitted\nfrom sklearn.base import BaseEstimator\nfrom sklearn.base import MetaEstimatorMixin\nfrom sklearn.base import clone\nfrom sklearn.base import is_classifier\nfrom sklearn.utils import Parallel, delayed, effective_n_jobs\nfrom sklearn.model_selection import check_cv\nfrom sklearn.model_selection._validation import _score\nfrom sklearn.metrics.scorer import check_scoring\nfrom sklearn.feature_selection.base import SelectorMixin\n\nfrom sklearn.utils import check_array, safe_mask\nfrom warnings import warn\n\n\ndef _rfe_single_fit(rfe, estimator, X, y, train, test, scorer):\n    \"\"\"\n    Return the score for a fit across one fold.\n    \"\"\"\n    X_train, y_train = _safe_split(estimator, X, y, train)\n    X_test, y_test = _safe_split(estimator, X, y, test, train)\n    return rfe._fit(\n        X_train, y_train, lambda estimator, features:\n        _score(estimator, X_test[:, features], y_test, scorer)).scores_\n\n\nclass RFE(BaseEstimator, MetaEstimatorMixin, SelectorMixin):\n    \"\"\"Feature ranking with recursive feature elimination.\n\n    Given an external estimator that assigns weights to features (e.g., the\n    coefficients of a linear model), the goal of recursive feature elimination\n    (RFE) is to select features by recursively considering smaller and smaller\n    sets of features. First, the estimator is trained on the initial set of\n    features and the importance of each feature is obtained either through a\n    ``coef_`` attribute or through a ``feature_importances_`` attribute.\n    Then, the least important features are pruned from current set of features.\n    That procedure is recursively repeated on the pruned set until the desired\n    number of features to select is eventually reached.\n\n    Read more in the :ref:`User Guide <rfe>`.\n\n    Parameters\n    ----------\n    estimator : object\n        A supervised learning estimator with a ``fit`` method that provides\n        information about feature importance either through a ``coef_``\n        attribute or through a ``feature_importances_`` attribute.\n\n    n_features_to_select : int or None (default=None)\n        The number of features to select. If `None`, half of the features\n        are selected.\n\n    step : int or float, optional (default=1)\n        If greater than or equal to 1, then ``step`` corresponds to the\n        (integer) number of features to remove at each iteration.\n        If within (0.0, 1.0), then ``step`` corresponds to the percentage\n        (rounded down) of features to remove at each iteration.\n\n    verbose : int, (default=0)\n        Controls verbosity of output.\n\n    Attributes\n    ----------\n    n_features_ : int\n        The number of selected features.\n\n    support_ : array of shape [n_features]\n        The mask of selected features.\n\n    ranking_ : array of shape [n_features]\n        The feature ranking, such that ``ranking_[i]`` corresponds to the\n        ranking position of the i-th feature. Selected (i.e., estimated\n        best) features are assigned rank 1.\n\n    estimator_ : object\n        The external estimator fit on the reduced dataset.\n\n    Examples\n    --------\n    The following example shows how to retrieve the 5 right informative\n    features in the Friedman #1 dataset.\n\n    >>> from sklearn.datasets import make_friedman1\n    >>> from sklearn.feature_selection import RFE\n    >>> from sklearn.svm import SVR\n    >>> X, y = make_friedman1(n_samples=50, n_features=10, random_state=0)\n    >>> estimator = SVR(kernel=\"linear\")\n    >>> selector = RFE(estimator, 5, step=1)\n    >>> selector = selector.fit(X, y)\n    >>> selector.support_ # doctest: +NORMALIZE_WHITESPACE\n    array([ True,  True,  True,  True,  True, False, False, False, False,\n           False])\n    >>> selector.ranking_\n    array([1, 1, 1, 1, 1, 6, 4, 3, 2, 5])\n\n    See also\n    --------\n    RFECV : Recursive feature elimination with built-in cross-validated\n        selection of the best number of features\n\n    References\n    ----------\n\n    .. [1] Guyon, I., Weston, J., Barnhill, S., & Vapnik, V., \"Gene selection\n           for cancer classification using support vector machines\",\n           Mach. Learn., 46(1-3), 389--422, 2002.\n    \"\"\"\n    def __init__(self, estimator, n_features_to_select=None, step=1,\n                 verbose=0):\n        self.estimator = estimator\n        self.n_features_to_select = n_features_to_select\n        self.step = step\n        self.verbose = verbose\n\n    @property\n    def _estimator_type(self):\n        return self.estimator._estimator_type\n\n    def fit(self, X, y):\n        \"\"\"Fit the RFE model and then the underlying estimator on the selected\n           features.\n\n        Parameters\n        ----------\n        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n            The training input samples.\n\n        y : array-like, shape = [n_samples]\n            The target values.\n        \"\"\"\n        return self._fit(X, y)\n\n    def _fit(self, X, y, step_score=None):\n        # Parameter step_score controls the calculation of self.scores_\n        # step_score is not exposed to users\n        # and is used when implementing RFECV\n        # self.scores_ will not be calculated when calling _fit through fit\n\n        X, y = check_X_y(X, y, \"csc\", force_all_finite=False)  # patched\n        # Initialization\n        n_features = X.shape[1]\n        if self.n_features_to_select is None:\n            n_features_to_select = n_features // 2\n        else:\n            n_features_to_select = self.n_features_to_select\n\n        if 0.0 < self.step < 1.0:\n            step = int(max(1, self.step * n_features))\n        else:\n            step = int(self.step)\n        if step <= 0:\n            raise ValueError(\"Step must be >0\")\n\n        support_ = np.ones(n_features, dtype=np.bool)\n        ranking_ = np.ones(n_features, dtype=np.int)\n\n        if step_score:\n            self.scores_ = []\n\n        # Elimination\n        while np.sum(support_) > n_features_to_select:\n            # Remaining features\n            features = np.arange(n_features)[support_]\n\n            # Rank the remaining features\n            estimator = clone(self.estimator)\n            if self.verbose > 0:\n                print(\"Fitting estimator with %d features.\" % np.sum(support_))\n\n            estimator.fit(X[:, features], y)\n\n            # Get coefs\n            if hasattr(estimator, 'coef_'):\n                coefs = estimator.coef_\n            else:\n                coefs = getattr(estimator, 'feature_importances_', None)\n            if coefs is None:\n                raise RuntimeError('The classifier does not expose '\n                                   '\"coef_\" or \"feature_importances_\" '\n                                   'attributes')\n\n            # Get ranks\n            if coefs.ndim > 1:\n                ranks = np.argsort(safe_sqr(coefs).sum(axis=0))\n            else:\n                ranks = np.argsort(safe_sqr(coefs))\n\n            # for sparse case ranks is matrix\n            ranks = np.ravel(ranks)\n\n            # Eliminate the worse features\n            threshold = min(step, np.sum(support_) - n_features_to_select)\n\n            # Compute step score on the previous selection iteration\n            # because 'estimator' must use features\n            # that have not been eliminated yet\n            if step_score:\n                self.scores_.append(step_score(estimator, features))\n            support_[features[ranks][:threshold]] = False\n            ranking_[np.logical_not(support_)] += 1\n\n        # Set final attributes\n        features = np.arange(n_features)[support_]\n        self.estimator_ = clone(self.estimator)\n        self.estimator_.fit(X[:, features], y)\n\n        # Compute step score when only n_features_to_select features left\n        if step_score:\n            self.scores_.append(step_score(self.estimator_, features))\n        self.n_features_ = support_.sum()\n        self.support_ = support_\n        self.ranking_ = ranking_\n\n        return self\n\n    @if_delegate_has_method(delegate='estimator')\n    def predict(self, X):\n        \"\"\"Reduce X to the selected features and then predict using the\n           underlying estimator.\n\n        Parameters\n        ----------\n        X : array of shape [n_samples, n_features]\n            The input samples.\n\n        Returns\n        -------\n        y : array of shape [n_samples]\n            The predicted target values.\n        \"\"\"\n        check_is_fitted(self, 'estimator_')\n        return self.estimator_.predict(self.transform(X))\n\n    @if_delegate_has_method(delegate='estimator')\n    def score(self, X, y):\n        \"\"\"Reduce X to the selected features and then return the score of the\n           underlying estimator.\n\n        Parameters\n        ----------\n        X : array of shape [n_samples, n_features]\n            The input samples.\n\n        y : array of shape [n_samples]\n            The target values.\n        \"\"\"\n        check_is_fitted(self, 'estimator_')\n        return self.estimator_.score(self.transform(X), y)\n\n    def _get_support_mask(self):\n        check_is_fitted(self, 'support_')\n        return self.support_\n\n    @if_delegate_has_method(delegate='estimator')\n    def decision_function(self, X):\n        \"\"\"Compute the decision function of ``X``.\n\n        Parameters\n        ----------\n        X : array-like or sparse matrix, shape = [n_samples, n_features]\n            The input samples. Internally, it will be converted to\n            ``dtype=np.float32`` and if a sparse matrix is provided\n            to a sparse ``csr_matrix``.\n\n        Returns\n        -------\n        score : array, shape = [n_samples, n_classes] or [n_samples]\n            The decision function of the input samples. The order of the\n            classes corresponds to that in the attribute `classes_`.\n            Regression and binary classification produce an array of shape\n            [n_samples].\n        \"\"\"\n        check_is_fitted(self, 'estimator_')\n        return self.estimator_.decision_function(self.transform(X))\n\n    @if_delegate_has_method(delegate='estimator')\n    def predict_proba(self, X):\n        \"\"\"Predict class probabilities for X.\n\n        Parameters\n        ----------\n        X : array-like or sparse matrix, shape = [n_samples, n_features]\n            The input samples. Internally, it will be converted to\n            ``dtype=np.float32`` and if a sparse matrix is provided\n            to a sparse ``csr_matrix``.\n\n        Returns\n        -------\n        p : array of shape = [n_samples, n_classes]\n            The class probabilities of the input samples. The order of the\n            classes corresponds to that in the attribute `classes_`.\n        \"\"\"\n        check_is_fitted(self, 'estimator_')\n        return self.estimator_.predict_proba(self.transform(X))\n\n    @if_delegate_has_method(delegate='estimator')\n    def predict_log_proba(self, X):\n        \"\"\"Predict class log-probabilities for X.\n\n        Parameters\n        ----------\n        X : array of shape [n_samples, n_features]\n            The input samples.\n\n        Returns\n        -------\n        p : array of shape = [n_samples, n_classes]\n            The class log-probabilities of the input samples. The order of the\n            classes corresponds to that in the attribute `classes_`.\n        \"\"\"\n        check_is_fitted(self, 'estimator_')\n        return self.estimator_.predict_log_proba(self.transform(X))\n    \nclass RFECV(RFE, MetaEstimatorMixin):\n    \"\"\"Feature ranking with recursive feature elimination and cross-validated\n    selection of the best number of features.\n\n    Read more in the :ref:`User Guide <rfe>`.\n\n    Parameters\n    ----------\n    estimator : object\n        A supervised learning estimator with a ``fit`` method that provides\n        information about feature importance either through a ``coef_``\n        attribute or through a ``feature_importances_`` attribute.\n\n    step : int or float, optional (default=1)\n        If greater than or equal to 1, then ``step`` corresponds to the\n        (integer) number of features to remove at each iteration.\n        If within (0.0, 1.0), then ``step`` corresponds to the percentage\n        (rounded down) of features to remove at each iteration.\n        Note that the last iteration may remove fewer than ``step`` features in\n        order to reach ``min_features_to_select``.\n\n    min_features_to_select : int, (default=1)\n        The minimum number of features to be selected. This number of features\n        will always be scored, even if the difference between the original\n        feature count and ``min_features_to_select`` isn't divisible by\n        ``step``.\n\n    cv : int, cross-validation generator or an iterable, optional\n        Determines the cross-validation splitting strategy.\n        Possible inputs for cv are:\n\n        - None, to use the default 3-fold cross-validation,\n        - integer, to specify the number of folds.\n        - An object to be used as a cross-validation generator.\n        - An iterable yielding train/test splits.\n\n        For integer/None inputs, if ``y`` is binary or multiclass,\n        :class:`sklearn.model_selection.StratifiedKFold` is used. If the\n        estimator is a classifier or if ``y`` is neither binary nor multiclass,\n        :class:`sklearn.model_selection.KFold` is used.\n\n        Refer :ref:`User Guide <cross_validation>` for the various\n        cross-validation strategies that can be used here.\n\n        .. versionchanged:: 0.20\n            ``cv`` default value of None will change from 3-fold to 5-fold\n            in v0.22.\n\n    scoring : string, callable or None, optional, (default=None)\n        A string (see model evaluation documentation) or\n        a scorer callable object / function with signature\n        ``scorer(estimator, X, y)``.\n\n    verbose : int, (default=0)\n        Controls verbosity of output.\n\n    n_jobs : int or None, optional (default=None)\n        Number of cores to run in parallel while fitting across folds.\n        ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.\n        ``-1`` means using all processors. See :term:`Glossary <n_jobs>`\n        for more details.\n\n    Attributes\n    ----------\n    n_features_ : int\n        The number of selected features with cross-validation.\n\n    support_ : array of shape [n_features]\n        The mask of selected features.\n\n    ranking_ : array of shape [n_features]\n        The feature ranking, such that `ranking_[i]`\n        corresponds to the ranking\n        position of the i-th feature.\n        Selected (i.e., estimated best)\n        features are assigned rank 1.\n\n    grid_scores_ : array of shape [n_subsets_of_features]\n        The cross-validation scores such that\n        ``grid_scores_[i]`` corresponds to\n        the CV score of the i-th subset of features.\n\n    estimator_ : object\n        The external estimator fit on the reduced dataset.\n\n    Notes\n    -----\n    The size of ``grid_scores_`` is equal to\n    ``ceil((n_features - min_features_to_select) / step) + 1``,\n    where step is the number of features removed at each iteration.\n\n    Examples\n    --------\n    The following example shows how to retrieve the a-priori not known 5\n    informative features in the Friedman #1 dataset.\n\n    >>> from sklearn.datasets import make_friedman1\n    >>> from sklearn.feature_selection import RFECV\n    >>> from sklearn.svm import SVR\n    >>> X, y = make_friedman1(n_samples=50, n_features=10, random_state=0)\n    >>> estimator = SVR(kernel=\"linear\")\n    >>> selector = RFECV(estimator, step=1, cv=5)\n    >>> selector = selector.fit(X, y)\n    >>> selector.support_ # doctest: +NORMALIZE_WHITESPACE\n    array([ True,  True,  True,  True,  True, False, False, False, False,\n           False])\n    >>> selector.ranking_\n    array([1, 1, 1, 1, 1, 6, 4, 3, 2, 5])\n\n    See also\n    --------\n    RFE : Recursive feature elimination\n\n    References\n    ----------\n\n    .. [1] Guyon, I., Weston, J., Barnhill, S., & Vapnik, V., \"Gene selection\n           for cancer classification using support vector machines\",\n           Mach. Learn., 46(1-3), 389--422, 2002.\n    \"\"\"\n    def __init__(self, estimator, step=1, min_features_to_select=1, cv='warn',\n                 scoring=None, verbose=0, n_jobs=None):\n        self.estimator = estimator\n        self.step = step\n        self.cv = cv\n        self.scoring = scoring\n        self.verbose = verbose\n        self.n_jobs = n_jobs\n        self.min_features_to_select = min_features_to_select\n\n    def transform(self, X):\n        \"\"\"Reduce X to the selected features.\n\n        Parameters\n        ----------\n        X : array of shape [n_samples, n_features]\n            The input samples.\n\n        Returns\n        -------\n        X_r : array of shape [n_samples, n_selected_features]\n            The input samples with only the selected features.\n        \"\"\"\n        X = check_array(X, dtype=None, accept_sparse='csr', force_all_finite=False)\n        mask = self.get_support()\n        if not mask.any():\n            warn(\"No features were selected: either the data is\"\n                 \" too noisy or the selection test too strict.\",\n                 UserWarning)\n            return np.empty(0).reshape((X.shape[0], 0))\n        if len(mask) != X.shape[1]:\n            raise ValueError(\"X has a different shape than during fitting.\")\n        return X[:, safe_mask(X, mask)]\n\n    def fit(self, X, y, groups=None):\n        \"\"\"Fit the RFE model and automatically tune the number of selected\n           features.\n\n        Parameters\n        ----------\n        X : {array-like, sparse matrix}, shape = [n_samples, n_features]\n            Training vector, where `n_samples` is the number of samples and\n            `n_features` is the total number of features.\n\n        y : array-like, shape = [n_samples]\n            Target values (integers for classification, real numbers for\n            regression).\n\n        groups : array-like, shape = [n_samples], optional\n            Group labels for the samples used while splitting the dataset into\n            train/test set.\n        \"\"\"\n        X, y = check_X_y(X, y, \"csr\", force_all_finite=False)\n\n        # Initialization\n        cv = check_cv(self.cv, y, is_classifier(self.estimator))\n        scorer = check_scoring(self.estimator, scoring=self.scoring)\n        n_features = X.shape[1]\n\n        if 0.0 < self.step < 1.0:\n            step = int(max(1, self.step * n_features))\n        else:\n            step = int(self.step)\n        if step <= 0:\n            raise ValueError(\"Step must be >0\")\n\n        # Build an RFE object, which will evaluate and score each possible\n        # feature count, down to self.min_features_to_select\n        rfe = RFE(estimator=self.estimator,\n                  n_features_to_select=self.min_features_to_select,\n                  step=self.step, verbose=self.verbose)\n\n        # Determine the number of subsets of features by fitting across\n        # the train folds and choosing the \"features_to_select\" parameter\n        # that gives the least averaged error across all folds.\n\n        # Note that joblib raises a non-picklable error for bound methods\n        # even if n_jobs is set to 1 with the default multiprocessing\n        # backend.\n        # This branching is done so that to\n        # make sure that user code that sets n_jobs to 1\n        # and provides bound methods as scorers is not broken with the\n        # addition of n_jobs parameter in version 0.18.\n\n        if effective_n_jobs(self.n_jobs) == 1:\n            parallel, func = list, _rfe_single_fit\n        else:\n            parallel = Parallel(n_jobs=self.n_jobs)\n            func = delayed(_rfe_single_fit)\n\n        scores = parallel(\n            func(rfe, self.estimator, X, y, train, test, scorer)\n            for train, test in cv.split(X, y, groups))\n\n        scores = np.sum(scores, axis=0)\n        scores_rev = scores[::-1]\n        argmax_idx = len(scores) - np.argmax(scores_rev) - 1\n        n_features_to_select = max(\n            n_features - (argmax_idx * step),\n            self.min_features_to_select)\n\n        # Re-execute an elimination with best_k over the whole set\n        rfe = RFE(estimator=self.estimator,\n                  n_features_to_select=n_features_to_select, step=self.step,\n                  verbose=self.verbose)\n\n        rfe.fit(X, y)\n\n        # Set final attributes\n        self.support_ = rfe.support_\n        self.n_features_ = rfe.n_features_\n        self.ranking_ = rfe.ranking_\n        self.estimator_ = clone(self.estimator)\n        self.estimator_.fit(self.transform(X), y)\n\n        # Fixing a normalization error, n is equal to get_n_splits(X, y) - 1\n        # here, the scores are normalized by get_n_splits(X, y)\n        self.grid_scores_ = scores[::-1] / cv.get_n_splits(X, y, groups)\n        return self","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1358f3451cdefb8a778ce2a6665b150b00e36eb1"},"cell_type":"markdown","source":"We will use a terrible training set to demonstrate how this works:"},{"metadata":{"trusted":true,"_uuid":"a405fb167940f86791cfe63cf7f86ee029ddd4ec"},"cell_type":"code","source":"sample_data = pd.read_csv('../input/training_set_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbca3472b96c52b655b1d31919117bdbab2372d9"},"cell_type":"code","source":"sample_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b73663e800e5f62e7b0028df503a67c5aab3f6bf"},"cell_type":"code","source":"X = sample_data.iloc[:, 1:-1].copy()\ny = sample_data['target']\n# manually sabotaging the data:\nX.loc[X['hostgal_specz'] == 0, 'hostgal_specz'] = np.inf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e843764874ccb8131d5e22456d62f3fabb168a84"},"cell_type":"code","source":"X.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e0dcb4b0a68165a791fedd849a9007441cb586c"},"cell_type":"code","source":"X.shape, y.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"13fbb5e5502388c5845d8545a6d8e89dcfbbb257"},"cell_type":"code","source":"clf_params = {\n        'boosting_type': 'gbdt',\n        'objective': 'multiclass',\n        'num_class': 14,\n        'metric': 'multi_logloss',\n        'learning_rate': 0.01,\n        'n_estimators': 500,\n        'verbose': -1,\n        'max_depth': 3,\n        'importance_type': 'gain',\n        'seed': 1111,\n        'n_jobs': -1,\n    }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b3b9e420ddf866123ee60291393ad0827b9dc858"},"cell_type":"code","source":"clf = lgb.LGBMClassifier(**clf_params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b288778d972797f40d8bfb4889b7bb3eed9de2b6"},"cell_type":"code","source":"rfe = RFE(clf, step=1, n_features_to_select=1, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd77bd6f2485fb65f31bc154a5ae6fe0a0bed43a"},"cell_type":"code","source":"rfe.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f061c2b3d2e4f76a7f754ecbf8b70f602a331dd"},"cell_type":"code","source":"# feature rankings\n{c: r for c, r in zip(X.columns, rfe.ranking_)}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9357305a15a2608318abf773b119dae9600f7ee"},"cell_type":"markdown","source":"We see that after the patching, RFE works with NaN and Inf just fine."},{"metadata":{"_uuid":"9397d34e6dd018d30d9eb7a0ecd91e752b708abd"},"cell_type":"markdown","source":"## 2. Passing parameters to fit() when SkLearn won't let you"},{"metadata":{"_uuid":"ea4cb61b1bba011aad37716c7b9082795e34538d"},"cell_type":"markdown","source":"Now that you have RFE working, you may want to let it use proper sample weights so your selected features will give you better performance in your final model. However, SkLearn sometimes stubbornly holds onto the simplicity of its API and would not allow you to pass any extra parameters to the call. If you want to do the following:"},{"metadata":{"trusted":true,"_uuid":"060e007afadf2ea2d69db2311e88a8b22901f6ad"},"cell_type":"code","source":"w = y.value_counts()\nsample_weight_map = {i: np.sum(w) / w[i] for i in w.index}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9df8fcd72530a3cb7acece205f882de3cc91db20"},"cell_type":"code","source":"rfe.fit(X, y, sample_weight=y.map(sample_weight_map))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51d2d433e8fa0512cda62a6412d251626a9d2d63"},"cell_type":"markdown","source":"it just won't happen. It sometimes actually makes sense, for example with CV estimators, each time the base estimator's `fit()` is called, the X and y might be different (depending on folds), and the sample_weight required may be different, so it does not make sense to pass the same weights to all of these method calls. But here, we really want to pass the same parameter to all of the `clf.fit()` calls. While we cannot change the SkLearn API, we can change the `clf.fit()` method to pass the sample weights automatically when SkLearn calls it. We need to insert something between `rfe.fit()` and SkLearn calling `clf.fit()` to smuggle in the `sample_weight` parameter. One seemingly easy solution is:"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"ec7c47ac3852847b945559aca930afc295b8c62f"},"cell_type":"code","source":"from functools import partial\nclf = lgb.LGBMClassifier(**clf_params)\nclf.fit = partial(clf.fit, sample_weight=y.map(sample_weight_map))\nrfe = RFE(clf, step=1, n_features_to_select=1, verbose=1)\nrfe.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ceb0c25581de8f372c87163d995217bfeb8b392"},"cell_type":"code","source":"{c: r for c, r in zip(X.columns, rfe.ranking_)}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a17bb6d9e672545bab1fa1deaa5744e478dc48b"},"cell_type":"markdown","source":"It appears to be working, but it is really deceiving. RFE uses `clone()` under the hood to make copies of the base estimator. Let us see if our modification survives the cloning:"},{"metadata":{"trusted":true,"_uuid":"8f73f9f3c74a27891af8f1f8a9ff8fd73711779a"},"cell_type":"code","source":"clf = lgb.LGBMClassifier(**clf_params)\nclf.fit = partial(clf.fit, sample_weight=y.map(sample_weight_map))\nstr(clf.fit)[:100]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e2ed2479ca9613284431484b6f3ba6d0457131c2"},"cell_type":"code","source":"str(clone(clf).fit)[:100]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0df3c4336998f1152be5c29035e2576f4f5568e0"},"cell_type":"markdown","source":"We see that in the cloned classifier, the method reverts to the original. Seems we have to modify the class for this to work. So can we do the same trick to the base class?"},{"metadata":{"trusted":true,"_uuid":"2368f237e549423a1b9e4eca5b40848e1f9cd354"},"cell_type":"code","source":"class NewLGBMClassifier(lgb.LGBMClassifier):\n    # override\n    def fit(self, X, y):\n        return super().fit(X, y, sample_weight=pd.Series(y).map(sample_weight_map))\n    # ^^^ sklearn actually converts y to numpy here already","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"67df48e1485efb6857fabdcd7d9e73cb57b6acb3"},"cell_type":"code","source":"clf = NewLGBMClassifier(**clf_params)\nrfe = RFE(clf, step=1, n_features_to_select=1, verbose=1)\nrfe.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6902f650452c20000112bb025f28ea13dfa5190b"},"cell_type":"code","source":"{c: r for c, r in zip(X.columns, rfe.ranking_)}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c4dc4ad6be18e47fab2da94197e16f7c7b72e1b"},"cell_type":"markdown","source":"We get a slightly different rankings, suggesting that something is different this time. The weights are actually being applied now. If we look at the final estimator from RFE:"},{"metadata":{"trusted":true,"_uuid":"f1408f99774f65fd1e0f15d0b30259ccf016f52a"},"cell_type":"code","source":"import inspect\ninspect.signature(rfe.estimator_.fit)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fbb5e71b030a3d14a22f26bd373e39ef50f5177c"},"cell_type":"code","source":"inspect.signature(lgb.LGBMClassifier.fit)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44ab0e1a08ff8907be49d4a6060bf48b3e902c5b"},"cell_type":"markdown","source":"We see that all the cloned estimators from RFE are the modified version now."},{"metadata":{"_uuid":"290a3d5f09eeb9b19404c254edc5e8220f7128ba"},"cell_type":"markdown","source":"As a bonus note, what if we really only want to modify that one instance? Does the original trick work? The code below will give you an error, indicating that the modifications are indeed applied:\n```\nclf = lgb.LGBMClassifier(**clf_params)\nclf.fit = partial(clf.fit, sample_weight=y.map(sample_weight_map))\nclf.fit(X.head(100), y.head(100))\n```\nHowever, if you really want a modified instance that can accept variable X, y and apply weights to them, you will need a different trick. We may attempt the following:"},{"metadata":{"trusted":true,"_uuid":"ecbaaa4c256f48f41c9448fff0d9a66c5d1e3dfa"},"cell_type":"code","source":"clf = lgb.LGBMClassifier(**clf_params)\n\ndef fit_override(self, X, y):\n    return clf.fit(X, y, sample_weight=y.map(sample_weight_map))\n\nclf.fit = fit_override\nclf.fit(X.head(100), y.head(100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d924b221b512d5c4443671ab3ce481c94f02d38"},"cell_type":"markdown","source":"But what we are really doing is assigning an ordinary function to the attribute `fit` of `clf`. It does not know it is supposed to be a method of `clf`. We should register the function as a method:"},{"metadata":{"trusted":true,"_uuid":"a91ca8cee7bb5dd5203b7930800adc51c124cfa6"},"cell_type":"code","source":"clf = lgb.LGBMClassifier(**clf_params)\n\ndef fit_override(self, X, y):\n    return self.fit(X, y, sample_weight=y.map(sample_weight_map))\n# or clf.fit(X, y, sample_weight=y.map(sample_weight_map))\n\nclf.fit = types.MethodType(fit_override, clf)\nclf.fit(X.head(100), y.head(100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"326193473f2cf704a3210dc659924bbb9264da1c"},"cell_type":"markdown","source":"But why is it still reporting an error? Because we are accidentally creating a recursion. When we call `clf.fit()`, it calls `fit_override` instead, then `fit_override` calls `clf.fit`, but `clf.fit` is `fit_override` now... So we are trying to pass `sample_weight` to `fit_override` which should not accept additional parameters other than X, y, hence the error. We can fix with the following trick:"},{"metadata":{"trusted":true,"_uuid":"bf9e1e3b35d152be9fbdba78d281b992c584c916"},"cell_type":"code","source":"clf = lgb.LGBMClassifier(**clf_params)\n\ndef get_override():\n    fit = clf.fit\n    # use closure to 'freeze' clf.fit\n    def fit_override(self, X, y):\n        return fit(X, y, sample_weight=y.map(sample_weight_map))\n    return fit_override\n\nclf.fit = types.MethodType(get_override(), clf)\nclf.fit(X.head(100), y.head(100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d33da4e2df693c58869555e253f2c01f091fb00c"},"cell_type":"markdown","source":"We see that this code is able to run, so the method freezing trick works! With this trick, you can do all sorts of method modding by inserting arbitrary code between the method call and the original method."},{"metadata":{"trusted":true,"_uuid":"25fe633d00bc777fe0a7ec4c42b7319eba310468"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}