{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"I've been trying to improve my preprocessing and modelling workflow and have successfully put together a [scikit-learn pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) to process the string feature and apply an XGBoost model. I wrote this notebook to demonstrate the use of a pipeline for linking together data preprocessing and model, rather than to present a high-scoring model in this month's Tabular Playground Series competition.  While there is amazing functionality and extensibility in scikit-learn pipelines, it is not as easy to use as I imagined it would be: for example, naming output columns, from e.g. `OneHotEncoder` is probably more difficult than it should be.\n\n# What is a pipeline?\n\nA pipeline is a series of processing and/or modelling steps bundled together into one object. From the [scikit-learn pipeline documentation](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html):\n\n> The purpose of the pipeline is to assemble several steps that can be cross-validated together while setting different parameters. \n\nQuite often we have a series of transformations (e.g. encoding categorical variables, power transforms for numerical variables, feature engineering, imputing missing values) that we'd like to apply to a dataset, and a candidate model (or models). Notice that preprocessing steps in scikit-learn all use `fit()` and `transform()` methods, and the machine learning estimators and models have similarly structured APIs, namely the `fit()` and `predict()` methods. The idea behind a pipeline is to use these similar APIs to link together transformations and models so that the output of one step feeds into the input of the next step in the pipeline.\n\n","metadata":{}},{"cell_type":"code","source":"from sklearn.pipeline import Pipeline\n\n# This displays pipelines as topological diagrams which are a bit more informative than the \n# python text __repr__\nfrom sklearn import set_config\nset_config(display='diagram')\n","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:18.494736Z","iopub.execute_input":"2022-07-31T07:39:18.495540Z","iopub.status.idle":"2022-07-31T07:39:18.918720Z","shell.execute_reply.started":"2022-07-31T07:39:18.495468Z","shell.execute_reply":"2022-07-31T07:39:18.918127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Benefits\n\nBenefits of using a pipeline:\n\n1. Eliminating duplicated code.\n\n    An important tenet of software engineering is [\"Don't repeat yourself\"](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself), or DRY. Copying and pasting code used to process training data to process the testing data is fraught with problems. Changes made to the data pre-processing workflow can be easily missed or errors made if you need to change the code in two (or more) places. Using a pipeline eliminates duplicated code by combining preprocessing steps into one class which can be easily instantiated again.\n\n2. Avoiding [data](https://machinelearningmastery.com/data-preparation-without-data-leakage/) [leakage](https://jfrog.com/community/data-science/be-careful-from-data-leakage/) by processing training and test data separately.\n\n    One (substandard) way to get around duplicated code is to combine the training and test data and apply the preprocessing to the combined dataset. I have done this before, and I see it quite commonly on kaggle, but it is [not recommended](https://community.alteryx.com/t5/Data-Science/Dealing-with-Data-Leakage/ba-p/827583). Briefly, preprocessing training and test data concurrently means test data information (e.g. distributions of features) is seen by the `fit()` method, and this will provide an overly optimistic test score, which ultimately leads to degraded performance on new predictions.\n\n3. Code readability.\n\n    By eliminating duplicated code and having everything in one place, readability of your code will be improved. \n4. Allows the possibility of tuning preprocessing choices for better test predictions.\n\n    Do you know what the effect is of the different preprocessing choices you've made on your model predictions? Testing the effect of preprocessing is typically done manually, by making a change and rerunning the notebook. This is slow and bulky to do, and it's very difficult to perform a proper workflow evaluation. Pipelines can automate the tuning and evaluation of your preprocessing workflow and help to answer questions like:\n    \n    1. Does particular preprocessing steps actually benefit the model?\n    2. For categorical encoding (e.g. OneHotEncoder), does grouping smaller infrequent classes help?\n    3. What parameters are best for numerical transforms (e.g. power transforms)?\n    4. Does dropping certain columns help predictions?\n    \n    \n# Data from the [Tabular Playground Series May 2022 competition](https://www.kaggle.com/competitions/tabular-playground-series-may-2022).\n\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\ntrain = pd.read_csv('/kaggle/input/tabular-playground-series-may-2022/train.csv', \n                    index_col='id')\ntest = pd.read_csv('/kaggle/input/tabular-playground-series-may-2022/test.csv', \n                   index_col='id')\n\n# This example data is used to demonstrate transformation of multiple categorical columns\n# As we are only transforming one column from train/test\nexample_categorial_data_columns = pd.DataFrame({'a1':['gds','fff','das','fbd','ggg'],\n                                                'a2':['xg','tt','fa','fd','tt']})\n\n# A small sample of the training data set for illustrative purposes:\ntrain_small = train.head(10).loc[:,['f_01','f_27','f_28']]\ntest_small = test.head(10).loc[:,['f_01','f_27','f_28']]","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:18.923683Z","iopub.execute_input":"2022-07-31T07:39:18.923879Z","iopub.status.idle":"2022-07-31T07:39:26.352054Z","shell.execute_reply.started":"2022-07-31T07:39:18.923856Z","shell.execute_reply":"2022-07-31T07:39:26.351444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Many wonderful EDAs have been produced for this competition, e.g.\n\n1. [AmbrosM](https://www.kaggle.com/code/ambrosm/tpsmay22-eda-which-makes-sense) as usual provided an amazing EDA.\n     \n2. [Ritika Gupta](https://www.kaggle.com/code/ritzig/feature-interaction-tutorial-pdp-shap-ensemble-mod) also discusses feature interactions.\n3. [Kelli Belcher](https://www.kaggle.com/code/kellibelcher/tps-may-2022-eda-lgbm-neural-networks) has an EDA with LGBM feature importance.\n4. [Naosher Mustakim](https://www.kaggle.com/code/naoshermustakim/comprehensive-eda-tps-may) also has a good EDA.\n5. I provided an [overview](https://www.kaggle.com/code/nnjjpp/eda-may-2022-exploring-the-string-feature-f-27) of the string feature (`f_27`)\n\nThere are many more EDAs, check the \"Code\" section of the competition.\n\nBriefly, the data consists of 30 features, one of which is a string of ten letters, fourteen of the remaining features are ordinal (integer-valued) variables, and the rest are continuous (real-valued) variables. The `train_small` sample data subset has the string feature and two continuous variables to demonstrate the pipeline.\n","metadata":{}},{"cell_type":"code","source":"train_small","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.353143Z","iopub.execute_input":"2022-07-31T07:39:26.353467Z","iopub.status.idle":"2022-07-31T07:39:26.367465Z","shell.execute_reply.started":"2022-07-31T07:39:26.353441Z","shell.execute_reply":"2022-07-31T07:39:26.366713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data preprocessing\n\nFor this notebook the only data preprocessing that I do is splitting of the string feature and encoding the letters. Obviously other preprocessing is possible, and feature engineering through the creation of aggregated features and interaction variables has been popular and successful in this competition. \n\n## Splitting and encoding of the string feature\n\nAs a list of strings the `f_27` feature can be easily separated using a python list comprehension:\n","metadata":{}},{"cell_type":"code","source":"def g(X):\n    return pd.DataFrame([list(x) for x in X])\n\nf_27_split = g(train_small['f_27'])\nf_27_split","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.369210Z","iopub.execute_input":"2022-07-31T07:39:26.369415Z","iopub.status.idle":"2022-07-31T07:39:26.384315Z","shell.execute_reply.started":"2022-07-31T07:39:26.369391Z","shell.execute_reply":"2022-07-31T07:39:26.383753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can then apply `OrdinalEncoder` to this to turn these letters into numerical codes, which is required for models like XGBoost.","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import OrdinalEncoder\n\nOrdinalEncoder().fit_transform(f_27_split)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.385418Z","iopub.execute_input":"2022-07-31T07:39:26.385844Z","iopub.status.idle":"2022-07-31T07:39:26.409704Z","shell.execute_reply.started":"2022-07-31T07:39:26.385799Z","shell.execute_reply":"2022-07-31T07:39:26.409071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create a transformer class\n\nIn order to use this transformation in a pipeline, we need to wrap the steps up into a transformer class. We inherit a few things from `BaseEstimator` and `TransformerMixin` (scikit-learn base classes). The actual splitting and encoding is performed in the `transform` method:","metadata":{}},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, TransformerMixin \n# the methods get_params() and set_params() are needed by \n# the pipeline and are inherited from BaseEstimator\n#\n# Inheriting from TransformerMixin provides us with a built-in \n# fit_transform() method\nclass SplitAndEncode(BaseEstimator,TransformerMixin): \n    def fit(self, X, y=None):\n        print(f'{self.__class__.__name__}(): fit called')\n        # Just save some information that could be useful in the transformation:\n        self.n_features_in = X.shape[1]\n        self.feature_names_in_ = X.columns\n        # Calculate the maximum length of string features, which is not\n        # used here but could be useful for processing variable length\n        # strings:\n        self.feature_lengths = X.aggregate(func=lambda y: max([len(x) for x in y])) \n        return self\n    def transform(self, X, y=None):\n        print(f'{self.__class__.__name__}(): transform called')\n        if self.n_features_in is None:\n            raise ValueError(f'Need to call {self.__class__.__name__}.fit() method first')\n        \n        #######################################\n        # Here is where the action takes place:\n        unencoded = np.column_stack([[list(x) for x in X.loc[:,col]] for i,col in \\\n                        enumerate(X.columns)])\n        from sklearn.preprocessing import OrdinalEncoder\n        oe = OrdinalEncoder()\n        return oe.fit_transform(unencoded)\n        #######################################\n        \n    def inverse_transform(self):\n        # We probably should define an inverse_transform method that combines the \n        # encoded columns back to a string\n        raise NotImplementedError\n\nSplitAndEncode().fit_transform(train_small[['f_27']])","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.410854Z","iopub.execute_input":"2022-07-31T07:39:26.411042Z","iopub.status.idle":"2022-07-31T07:39:26.426985Z","shell.execute_reply.started":"2022-07-31T07:39:26.411019Z","shell.execute_reply":"2022-07-31T07:39:26.426074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `column_stack` operation and enumeration of columns of `X` in the `transform` method above allows multiple categorical columns to be passed through. Although we only have one string feature in the competition data, applying the transformation to the example dataframe `example_categorical_data_columns` demonstrates this:","metadata":{}},{"cell_type":"code","source":"example_categorial_data_columns","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.428399Z","iopub.execute_input":"2022-07-31T07:39:26.429166Z","iopub.status.idle":"2022-07-31T07:39:26.444627Z","shell.execute_reply.started":"2022-07-31T07:39:26.429123Z","shell.execute_reply":"2022-07-31T07:39:26.443984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SplitAndEncode().fit_transform(example_categorial_data_columns)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.445987Z","iopub.execute_input":"2022-07-31T07:39:26.446566Z","iopub.status.idle":"2022-07-31T07:39:26.458000Z","shell.execute_reply.started":"2022-07-31T07:39:26.446521Z","shell.execute_reply":"2022-07-31T07:39:26.457186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So now we have the two steps, i.e. splitting the string and encoding the letters, wrapped up in a transformer class, which looks very similar to the usual preprocessing classes (e.g. `OrdinalEncoder`, `MinMaxScaler`) in scikit-learn.\n\nThe next step is to use a `ColumnTransformer`, which allows us to combine multiple column transformations (not done here), and allow the rest of the columns to pass through unchanged. [This article](https://machinelearningmastery.com/columntransformer-for-numerical-and-categorical-data/) at machinelearningmastery.com has a lot more detail on this and I found it very useful. As usual, [the scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) is also helpful.\n\nTo set up a `ColumnTransformer`, we make a list of transformers, each of which is specified by a name, the transformer instance and a list of columns. We also define what happens to columns that we haven't specified using the `remainder` keyword. The default is to ignore these columns, but we'd like to include them untransformed.","metadata":{}},{"cell_type":"code","source":"from sklearn.compose import ColumnTransformer\n\nct = ColumnTransformer(transformers=[('separate_and_encode_categorical', # name, can be anything\n                                      SplitAndEncode(), # transformer class instance\n                                      ['f_27']), # list of columns to be transformed\n                                     \n                                     # Can include additional transformers here...\n                                     \n                                    ],\n                       remainder='passthrough')\n\nct","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.459437Z","iopub.execute_input":"2022-07-31T07:39:26.460323Z","iopub.status.idle":"2022-07-31T07:39:26.479655Z","shell.execute_reply.started":"2022-07-31T07:39:26.460280Z","shell.execute_reply":"2022-07-31T07:39:26.479003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ct.fit_transform(train_small)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.480669Z","iopub.execute_input":"2022-07-31T07:39:26.480886Z","iopub.status.idle":"2022-07-31T07:39:26.494683Z","shell.execute_reply.started":"2022-07-31T07:39:26.480861Z","shell.execute_reply":"2022-07-31T07:39:26.493535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Notice that the columns have been reorded here: the columns in the transformed data follow the order specified in `ColumnTransformer`, with the transformed `f_27` variables appearing first, then the passthrough variables (i.e. `f_01` and `f_28`). This is fine for a model, but is far from ideal for us, as we have lost the column names and this could (probably will) cause problems and confusion later on.\n\nA common way of dealing with this lack of column names is to write over the columns in the dataframe with the transformed data in-place, e.g.:\n\n> <code>data.loc[:,columns_to_be_transformed] = OrdinalEncoder().fit_transform(data.loc[:,columns_to_be_transformed])</code>\n\nThis is problematic as we are either creating two versions of the same named dataset or creating temporary variables, neither of which is good data-science practice. Also, this workflow is unable (as far as I'm aware) to be incorporated into a pipeline. We need to somehow incorporate column renaming into the pipeline workflow.\n\n# Naming output feature columns within a pipeline\n\nPandas generally saves a copy of the column names going into transformers in the `feature_names_in_` attribute, and the API sometimes (but not always!) has a `get_feature_names_out` method that gives us a list of feature names corresponding to the output from the transformer. It looks like [newer releases of scikit-learn](https://scikit-learn.org/stable/whats_new/v1.1.html) are rolling out `get_feature_names_out` methods more consistently across estimators and transformers, but for the moment naming output columns is a bit tricky. I found a [question/answer on stack overflow that describes the problem](https://stackoverflow.com/questions/61079602/how-to-get-feature-names-using-a-column-transformer), and what follows is based loosely on this.\n\n## Implementing a feature names generator\n\nLet's implement a `get_feature_names_out` method for our transformer class by taking the input feature names and appending the letter position index to get 'f_27_0', 'f_27_1', and so on. Note that using the `feature_names_in_` attribute means we can pass through multiple columns into the transformer (if there was an additional string feature for example) and end up with meaningful names across different feature columns.\n\nBy creating a derived class from the `SplitAndEncode` class we [inherit](https://docs.python.org/3/tutorial/classes.html#inheritance) all the previous attributes and methods, so we can just define the new method:","metadata":{}},{"cell_type":"code","source":"class SplitAndEncodeWithNames(SplitAndEncode):\n    def get_feature_names_out(self):\n        names_out = []\n        for i, name_in in enumerate(self.feature_names_in_):\n            names_out += [f'{name_in}_{j}' for j in range(self.feature_lengths[i])]\n        return names_out\n\nsplit_transformer_with_names = SplitAndEncodeWithNames()\nsplit_transformer_with_names.fit(train_small[['f_27']])\nsplit_transformer_with_names.get_feature_names_out()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.496801Z","iopub.execute_input":"2022-07-31T07:39:26.497360Z","iopub.status.idle":"2022-07-31T07:39:26.506304Z","shell.execute_reply.started":"2022-07-31T07:39:26.497317Z","shell.execute_reply":"2022-07-31T07:39:26.505702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And let's try it on the example dataframe with two categorical columns:","metadata":{}},{"cell_type":"code","source":"split_transformer_with_names.fit(example_categorial_data_columns)\nsplit_transformer_with_names.get_feature_names_out()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.507229Z","iopub.execute_input":"2022-07-31T07:39:26.507575Z","iopub.status.idle":"2022-07-31T07:39:26.518010Z","shell.execute_reply.started":"2022-07-31T07:39:26.507538Z","shell.execute_reply":"2022-07-31T07:39:26.517452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So this works for single columns as well as multiple columns by iterating through the `feature_names_in_` attribute and the `feature_lengths` attribute that we set up in the `fit` method of the `SplitAndEncode` class.\n\n## Incorporating a get_feature_names_out method into ColumnTransformer\n\n`ColumnTransformer` does have a `get_feature_names_out` method but it [causes problems when upstream `get_feature_names_out` methods are not properly defined](https://johaupt.github.io/blog/columnTransformer_feature_names.html). In particular, the passthrough section of the `ColumnTransformer` doesn't provide feature names. The following class derived from `ColumnTransformer` gets output feature names from the transformers and appends the passthrough names:","metadata":{}},{"cell_type":"code","source":"class ColumnTransformerNamed(ColumnTransformer):\n    def get_feature_names_out(self):\n        names = []\n        for transformer in self.transformers_:\n            if transformer[0] == 'remainder':\n                if transformer[1] == 'passthrough':\n                    names += list(self.feature_names_in_[transformer[2]])\n                break\n            else:\n                names += transformer[1].get_feature_names_out()\n        return names\nct_named = ColumnTransformerNamed(transformers=[('separate_and_encode_categorical', \n                                                 SplitAndEncodeWithNames(), ['f_27'])],\n                                  remainder='passthrough')\n\nct_named","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.520339Z","iopub.execute_input":"2022-07-31T07:39:26.520739Z","iopub.status.idle":"2022-07-31T07:39:26.535408Z","shell.execute_reply.started":"2022-07-31T07:39:26.520632Z","shell.execute_reply":"2022-07-31T07:39:26.534836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_tr = ct_named.fit_transform(train_small)\nct_named.get_feature_names_out()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.536549Z","iopub.execute_input":"2022-07-31T07:39:26.536896Z","iopub.status.idle":"2022-07-31T07:39:26.546736Z","shell.execute_reply.started":"2022-07-31T07:39:26.536869Z","shell.execute_reply":"2022-07-31T07:39:26.545990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Final preprocessing pipeline step: convert to pandas dataframe and name columns\n\nThe `get_feature_names_out` methods are properly defined for the steps, so I create another transformer that converts the numpy array to a pandas dataframe and names the columns accordingly:","metadata":{}},{"cell_type":"code","source":"class rename(TransformerMixin):\n    def __init__(self, name_func):\n        self.name_func = name_func\n        pass\n    def fit(self, X, y=None):\n        self.names = self.name_func()\n        return self\n    def transform(self, X, y=None):\n        Xpd = pd.DataFrame(X)\n        Xpd.columns = self.names\n        return Xpd\n    def get_feature_names_out(self):\n        return self.name_func()","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.547932Z","iopub.execute_input":"2022-07-31T07:39:26.548366Z","iopub.status.idle":"2022-07-31T07:39:26.552722Z","shell.execute_reply.started":"2022-07-31T07:39:26.548334Z","shell.execute_reply":"2022-07-31T07:39:26.552233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can include this transformer as the next step in a pipeline, passing through a function that returns the column names (i.e. `get_feature_names_out`).\n\n# Assembling the preprocessing steps into a pipeline\n\nThe scikit-learn `Pipeline` [constructor syntax](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) is similar to the `ColumnTransformer` syntax. We specify a list of steps for the pipeline to perform. Alternatively, we can use the [`make_pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.make_pipeline.html) function, which differs from the constructor by not needing us to name the steps.","metadata":{}},{"cell_type":"code","source":"preprocessing_pipe = Pipeline(steps=(('tr', ct_named),\n                                     ('rename', rename(ct_named.get_feature_names_out))))\npreprocessing_pipe","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.553776Z","iopub.execute_input":"2022-07-31T07:39:26.554123Z","iopub.status.idle":"2022-07-31T07:39:26.591404Z","shell.execute_reply.started":"2022-07-31T07:39:26.554091Z","shell.execute_reply":"2022-07-31T07:39:26.590116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_transformed = preprocessing_pipe.fit_transform(train_small)\ntrain_transformed","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.592970Z","iopub.execute_input":"2022-07-31T07:39:26.593340Z","iopub.status.idle":"2022-07-31T07:39:26.637967Z","shell.execute_reply.started":"2022-07-31T07:39:26.593301Z","shell.execute_reply":"2022-07-31T07:39:26.637129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The pipeline we have made performs all the preprocessing we have specified, returns a pandas dataframe and has the columns nicely (and correctly) labelled. We can now apply the fitted pipeline to the test data:","metadata":{}},{"cell_type":"code","source":"%%time\npreprocessing_pipe.transform(test_small)","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.639301Z","iopub.execute_input":"2022-07-31T07:39:26.639788Z","iopub.status.idle":"2022-07-31T07:39:26.673444Z","shell.execute_reply.started":"2022-07-31T07:39:26.639746Z","shell.execute_reply":"2022-07-31T07:39:26.672597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"which works as expected. Note that this workflow (fitting on training data and only transforming the test data - notice that `fit` is not called on the test data) prevents data leakage as [described on Stack Overflow](https://stats.stackexchange.com/questions/267012/difference-between-preprocessing-train-and-test-set-before-and-after-splitting), which was one of our objectives in setting up the pipeline workflow.","metadata":{}},{"cell_type":"markdown","source":"# Fitting, transforming and predicting with the pipeline\n\nThe hard part is done and now we can use the pipeline to preprocess the data, fit the model, and use it to make predictions. To do this, just include a model as the final step. The preprocessed data will be used as an input into the model. Here we use XGBoost as an example.","metadata":{}},{"cell_type":"code","source":"from xgboost import XGBClassifier\nnum_xgb_ests = 150\npreprocessing_and_model_pipe = Pipeline(steps=(('tr', ct_named),\n                                               ('rename', \n                                                rename(ct_named.get_feature_names_out)),\n                                               ('xgboost', \n                                                XGBClassifier(n_estimators = num_xgb_ests,\n                                                             objective = 'binary:logistic'))))\npreprocessing_and_model_pipe","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.674545Z","iopub.execute_input":"2022-07-31T07:39:26.675139Z","iopub.status.idle":"2022-07-31T07:39:26.888024Z","shell.execute_reply.started":"2022-07-31T07:39:26.675094Z","shell.execute_reply":"2022-07-31T07:39:26.887189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calling `fit` on the pipeline runs `fit_transform` on the preprocessing steps and `fit` on the final model.","metadata":{}},{"cell_type":"code","source":"%%time\npreprocessing_and_model_pipe.fit(X = train.drop('target', axis=1),\n                                 y = train['target'])","metadata":{"execution":{"iopub.status.busy":"2022-07-31T07:39:26.889011Z","iopub.execute_input":"2022-07-31T07:39:26.889246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Predictions are made using the pipeline `predict` method:","metadata":{}},{"cell_type":"code","source":"%%time \npredictions = preprocessing_and_model_pipe.predict(X=test)\n\nsubmission = pd.DataFrame({'id': test.index,\n                           'target': predictions})\nsubmission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model evaluation\n\nFor model evaluation purposes, we plot the feature importances from the XGBoost model. We can use the column names from the `get_feature_names_out` method from within the pipeline via the `named_steps` attribute of the pipeline. ","metadata":{}},{"cell_type":"code","source":"# Function courtesy of Tyrion Lannister-lzy:\n# https://www.kaggle.com/code/tyrionlannisterlzy/xgboost-dnn-ensemble-lb-0-980\n\ndef plot_feature_importance(importance, names, model_type, max_features = 10):\n    #Create arrays from feature importance and feature names\n    feature_importance = np.array(importance)\n    feature_names = np.array(names)\n\n    #Create a DataFrame using a Dictionary\n    data={'feature_names':feature_names,'feature_importance':feature_importance}\n    fi_df = pd.DataFrame(data)\n\n    #Sort the DataFrame in order decreasing feature importance\n    fi_df.sort_values(by=['feature_importance'], ascending=False,inplace=True)\n    fi_df = fi_df.head(max_features)\n\n    #Define size of bar plot\n    plt.figure(figsize=(8,6))\n\n    #Plot Searborn bar chart\n    sns.barplot(x=fi_df['feature_importance'], y=fi_df['feature_names'])\n    #Add chart labels\n    plt.title(model_type + ' feature importance plot')\n    plt.xlabel('IMPORTANCE')\n    plt.ylabel('FEATURE NAMES')\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nplot_feature_importance(preprocessing_and_model_pipe.named_steps['xgboost'].feature_importances_,\n                        preprocessing_and_model_pipe.named_steps['rename'].get_feature_names_out(),\n                        f'XGBoost Classifier, {num_xgb_ests} estimators', max_features = 25)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Further directions\n\n- Implementation of the `inverse_transform` method\n- Allowing variable length strings (the `SplitAndEncode.feature_lengths` attribute can be used for this)\n- Incorporation of pipeline into a hyperparameter tuning workflow. Note that parameters for the internals of the pipeline (for example preprocessing steps) can be specified according to the [nested parameters](https://scikit-learn.org/stable/modules/compose.html#nested-parameters) section of the documentation.\n- Work out how to create new derived feature columns (aggregates, interactions, etc.)\n\n# Conclusions\n\n- Pipelines are an important and useful part of the data-science workflow, with the benefits of creating reproducible and more readable code, minimising data leakage, and offering the ability to tune parameters of the preprocessing step. \n- The use of a `ColumnTransformer` allows different columns to be processed differently.\n- Preserving column names in a pipeline can be a bit tricky, but is possible by using (and possibly modifying) `get_feature_names_out` methods.\n\nComments are welcome, please let me know if you have any suggestions on how to improve things, or if you have used pipelines in this competition or elsewhere.","metadata":{}}]}