{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## ***Intro***\n\nThe dataset of the \"*Home Credit - Credit Risk Model Stability*\" competition is definitely the largest dataset I have ever worked with in my young data science journey. There are just so many tables, rows and columns that we just can't reason with it the way we do with a single (base) table dataset. Extracting meaningful features for classification from all these tables seems even more challenging that the actual stability challenge of the competition (particularly when you are not an experienced Data engineer).\\\nThe competition is about to end in roughly one week, there is no more time for manual feature engineering (supposing any team even tried it).\\\nIn this Notebook I propose to explore the machine approach to creating features from raw data, **Automatic Feature Engineering**. AFE is the process of automatically creating hundreds or thousands of new features from a set of related tables in a fraction of the time as the manual approach. In this notebook, we will apply automated feature engineering to the Home Credit - Credit Risk Model Stability dataset using ***[Featuretools](http://www.featuretools.com/)*** an open source python framework for automated feature engineering and ***[Dask](http://www.dask.org)*** for parallel computing.","metadata":{}},{"cell_type":"markdown","source":"## ***Install Dask.distributed***\n\nAll the imported libraries should already be installed if your running this on kaggle except dask.distributed (that is intalled by the command below)","metadata":{}},{"cell_type":"code","source":"# You can comment this line if dask[distributed] is already installed in your environment\n%pip install dask[distributed]","metadata":{"execution":{"iopub.status.busy":"2024-05-20T14:47:16.569091Z","iopub.execute_input":"2024-05-20T14:47:16.569486Z","iopub.status.idle":"2024-05-20T14:47:35.478596Z","shell.execute_reply.started":"2024-05-20T14:47:16.569457Z","shell.execute_reply":"2024-05-20T14:47:35.476928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Imports***","metadata":{}},{"cell_type":"code","source":"# For data manipulation\nimport pandas as pd\n\n# To garbage collect the dataframes when their are not needed anymore\nimport gc\n# For retrieving files which have a certain pattern in their names (eg: train_applprev_1*.parquet)\nimport glob\n# To perform system operations\nimport os\nimport sys\n# To filter warnings\nimport warnings\n\n# For Automated Feature engineering \nimport featuretools as ft\n\n# For parallel computation\nimport dask.bag as db\nfrom dask.distributed import Client, progress","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-05-20T15:59:22.675830Z","iopub.execute_input":"2024-05-20T15:59:22.676312Z","iopub.status.idle":"2024-05-20T15:59:29.025568Z","shell.execute_reply.started":"2024-05-20T15:59:22.676277Z","shell.execute_reply":"2024-05-20T15:59:29.024314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Functions***","metadata":{}},{"cell_type":"code","source":"def partition_dataset(root_dir, entity_names, length_partitions):\n    \"\"\"\n    Partitions the dataset based on case_id values in the base table and entity names.\n    \n    Args:\n        root_dir (str): Path to the directory containing the original tables (Competition dataset)\n        entity_names (list): List of entity names (e.g., [\"static_0\", \"applprev_1\", \"base\", ...]).\n        length_partitions (int): Number of case_ids per partition.\n    \n    Returns:\n        None: The function creates subsets of each entity (per case_id) in each partition.\n    \"\"\"\n    # Read base table\n    base_df = pd.read_parquet(root_dir + \"base.parquet\")\n    case_id_subsets = [base_df[\"case_id\"].iloc[i:i+length_partitions] for i in range(0, base_df.shape[0], length_partitions)]\n    # Delete the table and keep only the subset ids since base will also be partitionned\n    del base_df\n    gc.collect()\n    \n    # Iterate through partitions and entities\n    for partition_id, case_id_subset in enumerate(case_id_subsets):\n        # Make the directory for partition\n        directory = f\"/kaggle/working/partitions/partition{partition_id + 1}\"\n        if os.path.exists(directory):\n            continue\n        \n        os.makedirs(directory)\n        for entity_name in entity_names:\n            # Check for single or multiple table entity\n            entity_files = glob.glob(f\"{root_dir}{entity_name}*.parquet\")\n            \n            if len(entity_files) == 1:\n                # Single table entity\n                entity_df = pd.read_parquet(entity_files[0])\n                subset_df = entity_df[entity_df[\"case_id\"].isin(case_id_subset)]\n                subset_df.to_parquet(f\"{directory}/{entity_name}.parquet\")\n                del entity_df, subset_df\n                gc.collect()\n            else:\n                # Multiple table entity\n                entity_df = pd.read_parquet(entity_files[0])\n                subset_df = entity_df[entity_df[\"case_id\"].isin(case_id_subset)]\n                for file_path in entity_files[1:]:\n                    entity_df = pd.read_parquet(file_path)\n                    subset_continue = entity_df[entity_df[\"case_id\"].isin(case_id_subset)]\n                    if not subset_continue.empty:\n                        subset_df = pd.concat([subset_df, subset_continue])\n                # Save subset and delete intermetiade tables.\n                # Note that this line will save the subset_df even if it's empty (meaning there is no info \n                # regarding the specific subset of case_ids in that particular entity, same for single table entities)\n                # Each EntitySet should have the same number of entities in order to calculate the same features for each partition \n                subset_df.to_parquet(f\"{directory}/{entity_name}.parquet\")\n                del entity_df, subset_df, subset_continue\n                gc.collect()\n        print(f\"Done parition{partition_id}, saved in {directory}\")\n\ndef read_and_init_ww(path, name, logical_types=None):\n    \"\"\"\n    Reads a DataFrame from a Parquet file, initializes it with Woodwork, and performs basic data typing and semantic tagging.\n    \n    Args:\n        path (str): Path to the Parquet file containing the DataFrame.\n        name (str): Name to assign to the DataFrame within Woodwork.\n        logical_types (dict): A dictionary with the logical types of the different columns in the EntitySet that was used when building the features (Default to None)\n        \n    Returns:\n        pandas.DataFrame: The loaded and Woodwork-initialized DataFrame.\n    \n    Raises:\n        ValueError: If the provided path does not point to a valid Parquet file.\n  \"\"\"\n    try:\n        df = pd.read_parquet(path)\n    except FileNotFoundError:\n        raise ValueError(f\"Invalid path: {path}\")\n    \n    if name == \"base\":\n        #Drop MONTH because it can be calculated with \"date_decision\"\n        # And target because because we don't want any features to be derived from the target to predict\n        df.drop(columns=[\"MONTH\", \"target\"], inplace=True)\n        # Init woodwork DataFrame\n        df.ww.init(name=name, index=\"case_id\")\n        print(\"Initialized dataframe $base, target and MONTH dropped\")\n        return df\n    \n    # Set a unique index\n    df.reset_index(drop=True, inplace=True)\n    df[\"id\"] = df.index\n    if logical_types:\n        logical_types = logical_types[name]\n        # This block of code has been added in response to a ValueError that occurred when runing the Bag for the first time\n        for col, l_type in logical_types.items():\n            if str(l_type) == \"Boolean\":\n                logical_types[col] = \"BooleanNullable\"\n                print(f\"Changed logical type of column {col} from Boolean to BooleanNullable\")\n        df.ww.init(name=name, index=\"id\", logical_types=logical_types)\n    else:\n        df.ww.init(name=name, index=\"id\")\n        \n    # Convert integer representations of booleans to true Booleans and Add semantic tags to birth dates\n    for col in df.columns:\n        # Woodwork should properly infer the correct logical types for all the columns but this is a precaution\n        if not logical_types: # Running for the reference partition\n            if set(df[col].unique()) == {0,1} and str(df.ww.logical_types[col]) == \"Integer\":\n                df.ww.set_types(logical_types={col: \"Boolean\"})\n                print(\"Type Integer changed to true logical type Bool\")\n            if col[-1] == \"P\" and str(df.ww.logical_types[col]) != \"Double\":\n                df.ww.set_types(logical_types={col: \"Double\"})\n            if col[-1] == \"M\" and str(df.ww.logical_types[col]) != \"Categorical\":\n                df.ww.set_types(logical_types={col: \"Categorical\"})\n            if col[-1] == \"A\" and str(df.ww.logical_types[col]) != \"Double\":\n                df.ww.set_types(logical_types={col: \"Double\"})\n            if col[-1] == \"D\" and str(df.ww.logical_types[col])!= \"Datetime\":\n                df.ww.set_types(logical_types={col: \"Datetime\"})  \n        # Add semantic_tags to birth dates columns    \n        if col.__contains__(\"birth\"):\n            df.ww.add_semantic_tags(semantic_tags = {col: \"date_of_birth\"})\n            print(f\"Added semantic tag $date_of_birth to col {col}\")\n    print(f\"Initialized dataframe ${name}, no columns dropped\")\n    return df\n\ndef add_relationships(es, children_names, parent_name=\"base\"):\n    \"\"\"\n    Adds a relationship between a parent Entity and its childrens within a Featuretools EntitySet.\n    \n    Args:\n        es (featuretools.EntitySet): The Featuretools EntitySet to add the relationship to.\n        child_names (list[str]): The names of the child DataFrames/Entities within the EntitySet.\n        parent_name (str, optional): The name of the parent DataFrame/Entity (defaults to \"base\").\n    \n    Returns:\n        featuretools.EntitySet: The modified EntitySet with the added relationship.\n    \"\"\"\n    for child_name in children_names:\n        es = es.add_relationship(\n        parent_dataframe_name=parent_name,\n        parent_column_name=\"case_id\",\n        child_dataframe_name=child_name,\n        child_column_name=\"case_id\")\n    return es\n\ndef entitySet_from_part(path_part, dfs_logical_types=None):\n    \"\"\"\n    This function constructs a Featuretools EntitySet from a partition path.\n    \n    Args:\n        path_part (str): The path of the partition (should contains all 17 entities). \n        dfs_logical_types: The logical types in the EntitySet that has been used to build the feature (Default to None)\n    Returns:\n        dict: A dictionary containing two keys:\n        - \"es\" (Featuretools.EntitySet): The constructed Featuretools EntitySet containing all entities and their relationships.\n        - \"num_partition\" (str): The extracted partition number from the provided path.\n    \"\"\"\n    # Filter warnings\n    warnings.filterwarnings('ignore')\n    \n    entities_paths = glob.glob(path_part + \"/*\")\n    entities_names = [path.split(\"/\")[-1].split(\".\")[0] for path in entities_paths]\n    children_names = entities_names.copy()\n    children_names.remove(\"base\")\n    dfs = [read_and_init_ww(path=path, name=name, logical_types=dfs_logical_types) for path, name in zip(entities_paths, entities_names)]\n    es = ft.EntitySet(id=\"home_credit\")\n    for df in dfs:\n        es.add_dataframe(dataframe=df)\n    es = add_relationships(es=es, children_names=children_names)\n    return {\"es\": es, \"num_partition\": path_part[20:]}\n\ndef get_unknownType_columns(entityset):\n    \"\"\"\n    Finds columns with the logical type \"Unknown\" in a Featuretools EntitySet.\n    \n    Args:\n        entityset (Featuretools.EntitySet): The EntitySet to analyze.\n    \n    Returns:\n        list: A list of tuples containing (entity_name, column_name) for unknown types.\n    \"\"\"\n    unknown_columns = []\n    for entity in entityset.dataframes:\n        for column in entity.columns:\n            if str(entity.ww.logical_types[column]) == \"Unknown\":\n                unknown_columns.append((entity.ww.name, column))\n    return unknown_columns\n\ndef feature_matrix_from_es(es_dict, feature_defs, return_fm = False):\n    \"\"\"\n    Calculates and saves a feature matrix from an EntitySet.\n    \n    Args:\n        es_dict (dict): Output dictionary from entitySet_from_part.\n        feature_defs (list): List of Featuretools feature definitions.\n        return_fm (bool, optional): If True, return the feature matrix (default: False, saves to Parquet file).\n    \n    Returns:\n        None (default) or pandas.DataFrame:\n    \"\"\"\n    # Filter warnings\n    warnings.filterwarnings('ignore')\n    \n    # Extract the entityset from the dictionary returned by entitySet_from_part\n    es = es_dict['es']\n    # Calculate the feature matrix and save\n    feature_matrix = ft.calculate_feature_matrix(feature_defs, entityset=es, n_jobs=1, chunk_size = es['base'].shape[0], verbose=True)\n    print(\"Done feature matrix \" + es_dict[\"num_partition\"])\n    # Save fm to parquet\n    feature_matrix.to_parquet('/kaggle/working/feature_matrices/partition' + es_dict[\"num_partition\"] + \"_fm.parquet\")\n    print(\"Saved feature matrix \" + es_dict[\"num_partition\"])\n    \n    if return_fm:\n        return feature_matrix\n","metadata":{"execution":{"iopub.status.busy":"2024-05-20T15:59:29.029168Z","iopub.execute_input":"2024-05-20T15:59:29.029952Z","iopub.status.idle":"2024-05-20T15:59:29.070676Z","shell.execute_reply.started":"2024-05-20T15:59:29.029908Z","shell.execute_reply":"2024-05-20T15:59:29.069375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Partitioning the dataset***\n\n**First and foremost we need to repartition the competion dataset**\\\nWe will divide the dataset by training examples and entities, currently the dataset for this competion contains 17 \"entities\", some of which spread in multiple tables like **credit_bureau_a_1**. We want independent partitions containing all the informations about a subset of ***n*** training examples or ***case_ids***\\\nCreating partitions that contain all the entities necessary to create an EntitySet (one entity == one table) for a subset of the training examples will allow us to calculate features matrices for the different subsets independently and simultaneously on different cpu cores with dask bags","metadata":{}},{"cell_type":"code","source":"!mkdir partitions\n%ls","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_root = \"/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_\"\nentity_names = [\"base\", \"static_0\", \"static_cb_0\", \"applprev_1\", \"other_1\", \"tax_registry_a_1\",\"tax_registry_b_1\", \n            \"tax_registry_c_1\", \"credit_bureau_a_1\", \"credit_bureau_b_1\", \"deposit_1\", \"person_1\", \"debitcard_1\", \n            \"applprev_2\", \"person_2\", \"credit_bureau_a_2\", \"credit_bureau_b_2\"]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Partition the dataset, 12,000 case_ids per partition (This operation will take a while)\npartition_dataset(root_dir=train_root, entity_names=entity_names, length_partitions=12000)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Deep Features Synthesis***\n\nThe automatic creation of features is a relatively recent concept in the field of data science but with undeniable potential to reduce the workload of data scientists, particularly at the data level. The concept was first formalized by James Max Kanter and Kalyan Veeramachaneni in their 2015 paper (\"Deep feature synthesis: Towards automating data science endeavors\"), where they proposed an algorithm to automatically generate predictors for relational datasets. This is the Deep Feature Synthesis algorithm (Kanter and Veeramachaneni, 2015).\n\nThe algorithm follows the relationships between different tables and a base table, then sequentially applies mathematical functions along these paths to create features. The algorithm takes as input an interconnected set of entities/tables.\n\nIn the paper, the authors described two types of relations between entities: forward and backward, and three types of features for an entity based on the types of relations it maintains with other entities: entity features (efeat), direct features (dfeat), and relational features (rfeat).\n\nIn a forward relation, an instance of an entity E<sup>L</sup> is associated with a single instance of another entity E<sup>k</sup>. The backward relation is the relation of an instance i in E<sup>k</sup> to all instances m = {1 . . . M} in E<sup>L</sup> with which it has a forward relation with k.\n\nA trivial example is that of a \"parent-child\" relation where a child basically has only one parent while the parent can have many children. Scientists have largely adopted this terminology.\n\n- **Entity Features (efeat)**: These are features directly related to the entity (variables in the base table of the entity) such as gender, age, income, etc. Depending on the variable to be predicted, this type of feature could even be used directly as predictors. Transformations can also be applied to them, for example: transforming a timestamp into day, month, year, and hour or calculating age from the date of birth.\n\n- **Direct Features (dfeat)**: Direct features are applied to forward relations (Child -> Parent). In these, features of entity i ∈ E<sup>k</sup> (child) are directly transferred as characteristics for entity m ∈ E<sup>L</sup> (parent). An example could be the price of a product becoming an attribute in the orders table (a feature for the Order entity).\n\n- **Relational Features (rfeat)**: Relational features are applied to backward relations (Parent -> Children). They are derived for an instance i of entity Ek by applying a mathematical function to x<sup>L</sup><sub>:,j|ek =i</sub>, which is a collection of values for a feature j in entity E<sup>L</sup>, assembled by extracting all values of j in entity E<sup>L</sup> where the identifier of E<sup>k</sup> is e<sup>k</sup> = i. This transformation is given by the formula: x<sup>k</sup><sub>i,j’</sub>= rfeat(x<sup>L</sup><sub>:,j|ek =i</sub>). Some examples of rfeat functions are min, max, and count.\n\nThe goal of the Deep Feature Synthesis algorithm is to find and compute the dfeats and rfeats for a target entity E<sup>k</sup>, add them to the E<sup>k</sup> table, and calculate the efeats on all of this. \n\nIn the case where a child entity of the target entity has its own children, we can recursively generate features using the same sequence described above. Recursion can end when a certain depth is reached or there are no more children. This [image](https://drive.google.com/file/d/13vDhyJVkxBeBMiG_UxRSpydSIXWDqfLz/view?usp=sharing) illustrates an example application of the algorithm\n\nFeaturetools already implements these concepts, the following cells create an EntitySet from one of the partitions created above and run dfs to get 2526 new features derived from the given entityset.","metadata":{}},{"cell_type":"code","source":"# We will use only one entitySet to run dfs and get the features that we will later calculate for all partitions\n# In theory you can use whatever partition you want to build this EntitySet, but in some partitions there are tables that don't have any row\n# these tables' columns (T or L unspecified transforms) will be inferred by woodwork with the datatypes \"unknown\" \n# which will impact the features generated by DFS. To avoid this I recommand choosing a partition where all the \n# tables have at least one row (here I found partition5) after some trials\nes_dict = entitySet_from_part(\"partitions/partition5\")","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:39:35.837744Z","iopub.execute_input":"2024-05-20T04:39:35.838155Z","iopub.status.idle":"2024-05-20T04:40:28.193112Z","shell.execute_reply.started":"2024-05-20T04:39:35.838126Z","shell.execute_reply":"2024-05-20T04:40:28.191696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"es = es_dict[\"es\"]\nes","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:40:28.195663Z","iopub.execute_input":"2024-05-20T04:40:28.196103Z","iopub.status.idle":"2024-05-20T04:40:28.207053Z","shell.execute_reply.started":"2024-05-20T04:40:28.196069Z","shell.execute_reply":"2024-05-20T04:40:28.205421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"es[\"base\"]","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:40:28.208991Z","iopub.execute_input":"2024-05-20T04:40:28.209508Z","iopub.status.idle":"2024-05-20T04:40:28.231925Z","shell.execute_reply.started":"2024-05-20T04:40:28.209462Z","shell.execute_reply":"2024-05-20T04:40:28.230721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Even if we have an entityset that have no empty dataframes, we still can have some columns with no values\n# Given that some columns in the original Dataset have 90+% null values, it's definitely possible and these columns...\n# will be infered by woodwork as unknown (though we should have significantly less unknwon columns) so we will have to type them manually\nunknowns = get_unknownType_columns(es)\nunknowns","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:40:28.235682Z","iopub.execute_input":"2024-05-20T04:40:28.236178Z","iopub.status.idle":"2024-05-20T04:40:28.266256Z","shell.execute_reply.started":"2024-05-20T04:40:28.236144Z","shell.execute_reply":"2024-05-20T04:40:28.264648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fortunately we have just one unknown type columns so we can easily set his type manually after checking the original dataset\nes['credit_bureau_b_1'].ww.set_types(logical_types={'periodicityofpmts_997L': \"Categorical\"})","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:40:28.268102Z","iopub.execute_input":"2024-05-20T04:40:28.268541Z","iopub.status.idle":"2024-05-20T04:40:28.277840Z","shell.execute_reply.started":"2024-05-20T04:40:28.268507Z","shell.execute_reply":"2024-05-20T04:40:28.276476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the logical types of all the columns in the EntitySet that we will use to build the features\n# This will be passed to the entitySet_from_part function so to ensure that the same columns \n# from different partitions will have the same logical types \nlogical_types = {}\n\nfor df in es.dataframes:\n    logical_types[df.ww.name] = df.ww.logical_types\n\nlogical_types[\"base\"]","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:40:28.279603Z","iopub.execute_input":"2024-05-20T04:40:28.280049Z","iopub.status.idle":"2024-05-20T04:40:28.294214Z","shell.execute_reply.started":"2024-05-20T04:40:28.280015Z","shell.execute_reply":"2024-05-20T04:40:28.292680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Visualize the EntitySet and the relationships between the tables\n# Should output a pretty large graph\nes.plot()","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:40:28.296369Z","iopub.execute_input":"2024-05-20T04:40:28.297386Z","iopub.status.idle":"2024-05-20T04:40:28.617695Z","shell.execute_reply.started":"2024-05-20T04:40:28.297340Z","shell.execute_reply":"2024-05-20T04:40:28.616470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# To get an estimation of the size of one EntitiSet\nestimated_size = sys.getsizeof(es)\nprint(f\"Estimated memory usage of EntitySet: {estimated_size/1e9} GB\")","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:41:04.764161Z","iopub.execute_input":"2024-05-20T04:41:04.764664Z","iopub.status.idle":"2024-05-20T04:41:04.815256Z","shell.execute_reply.started":"2024-05-20T04:41:04.764628Z","shell.execute_reply":"2024-05-20T04:41:04.813956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# List the built-in primitives in Featuretools\nprimitives = ft.list_primitives()\nprimitives[primitives['type'] == 'aggregation'].head()","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:41:08.238141Z","iopub.execute_input":"2024-05-20T04:41:08.238642Z","iopub.status.idle":"2024-05-20T04:41:08.273327Z","shell.execute_reply.started":"2024-05-20T04:41:08.238607Z","shell.execute_reply":"2024-05-20T04:41:08.271617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"primitives[primitives['type'] == 'transform'].head()","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:41:13.968119Z","iopub.execute_input":"2024-05-20T04:41:13.969444Z","iopub.status.idle":"2024-05-20T04:41:13.986880Z","shell.execute_reply.started":"2024-05-20T04:41:13.969375Z","shell.execute_reply":"2024-05-20T04:41:13.985707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Selection a set of aggregation and transformation primitives to run DFS\nagg_primitives = [\"sum\", \"num_unique\", \"std\", \"max\", \"skew\", \"min\", \"mean\", \"count\", \"percent_true\", \"mode\", \"max_min_delta\", \"entropy\"]\ntrans_primitives = [\"year\", \"month\", \"weekday\", \"season\", \"is_null\", \"percentile\", \"age\"]\n\n# DFS with specified primitives\n# This will only returns the names of the features without calculating the actual feature matrix\nfeature_names = ft.dfs(entityset=es, target_dataframe_name='base',\n                       trans_primitives=trans_primitives,\n                       agg_primitives=agg_primitives, \n                       max_depth=1, n_jobs=-1, verbose=1,\n                       features_only=True)","metadata":{"execution":{"iopub.status.busy":"2024-05-20T00:12:17.591050Z","iopub.execute_input":"2024-05-20T00:12:17.591571Z","iopub.status.idle":"2024-05-20T00:12:18.391335Z","shell.execute_reply.started":"2024-05-20T00:12:17.591535Z","shell.execute_reply":"2024-05-20T00:12:18.390320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 2526 features were built, let's take a look at some of them\nlen(feature_names), feature_names[:10]","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:41:59.677796Z","iopub.execute_input":"2024-05-20T04:41:59.678309Z","iopub.status.idle":"2024-05-20T04:42:01.134934Z","shell.execute_reply.started":"2024-05-20T04:41:59.678277Z","shell.execute_reply":"2024-05-20T04:42:01.133324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the feature names\nft.save_features(feature_names, 'features.txt')","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:42:23.273160Z","iopub.execute_input":"2024-05-20T04:42:23.273601Z","iopub.status.idle":"2024-05-20T04:42:23.560309Z","shell.execute_reply.started":"2024-05-20T04:42:23.273552Z","shell.execute_reply":"2024-05-20T04:42:23.558881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Calculate the feature matrices with Dask***\n\nIn the previous step we have just defined the features with DFS but we haven't calculated the actual values of each feature for each case_id in the subset (partition). That's where the problems start, as I said earlier the dataset for this competition is huge. Even with the supermachine offered by kaggle calculating the 2526 featues for one partition should take about 20 minutes, so approximately 42 hours to complete this operation on all the partions (20 * 128) / 60 ~= 42.\\\nWe definitely need a better approach and Dask gives us this second option. By parallelizing the operations needed to calculate the feature matrix for one partition (Creating entityset from partition and calculating feature matrix from ES) Dask will be able to load different EntiySets on all the available cpu cores and calculate their feature matrices independently and simultaneously on those cores.\\\nIf you run this on kaggle you should have 4 cores. The tasks must be small enough that they don't exhaust the memory of an individual worker. Basically the amount of RAM allocated to each cpu core is the total amount of available RAM divided by the number of cores, 8 GB with a kaggle notebook, that is largely enough for the operations needed to create an EntitySet from a partition and calculate the feature matrix for this ES","metadata":{}},{"cell_type":"code","source":"# Create the directory to store the feature matrices\n!mkdir feature_matrices\n%ls","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:42:38.035328Z","iopub.execute_input":"2024-05-20T04:42:38.035797Z","iopub.status.idle":"2024-05-20T04:42:39.165409Z","shell.execute_reply.started":"2024-05-20T04:42:38.035765Z","shell.execute_reply":"2024-05-20T04:42:39.163269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nproc --all # display the number of cores of the machine","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:42:47.374740Z","iopub.execute_input":"2024-05-20T04:42:47.375269Z","iopub.status.idle":"2024-05-20T04:42:48.531240Z","shell.execute_reply.started":"2024-05-20T04:42:47.375223Z","shell.execute_reply":"2024-05-20T04:42:48.529245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Free up memory before\ndel es_dict, es\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:42:50.141970Z","iopub.execute_input":"2024-05-20T04:42:50.142676Z","iopub.status.idle":"2024-05-20T04:42:50.354307Z","shell.execute_reply.started":"2024-05-20T04:42:50.142638Z","shell.execute_reply":"2024-05-20T04:42:50.352620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use all 4 cores\nclient = Client(processes = True)\nclient.ncores()","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:42:59.920607Z","iopub.execute_input":"2024-05-20T04:42:59.921025Z","iopub.status.idle":"2024-05-20T04:43:03.577861Z","shell.execute_reply.started":"2024-05-20T04:42:59.920997Z","shell.execute_reply":"2024-05-20T04:43:03.576151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# list paths to all the partitions\npaths = glob.glob(\"partitions/*\")\n\npaths[:4], len(paths)","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:43:04.438622Z","iopub.execute_input":"2024-05-20T04:43:04.439073Z","iopub.status.idle":"2024-05-20T04:43:04.450024Z","shell.execute_reply.started":"2024-05-20T04:43:04.439040Z","shell.execute_reply":"2024-05-20T04:43:04.448548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a bag object (Note that I had to run two bags with different list of paths baecause the session had expired)\nbag = db.from_sequence(paths)\n# Map entitySet_from_part function\nbag = bag.map(entitySet_from_part, dfs_logical_types=logical_types)\n# Map feature_matrix_from_es function\nbag = bag.map(feature_matrix_from_es, feature_defs = feature_names)\nbag","metadata":{"execution":{"iopub.status.busy":"2024-05-20T04:43:36.555304Z","iopub.execute_input":"2024-05-20T04:43:36.556278Z","iopub.status.idle":"2024-05-20T04:46:15.583037Z","shell.execute_reply.started":"2024-05-20T04:43:36.556213Z","shell.execute_reply":"2024-05-20T04:46:15.582008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bag = bag.persist()\nprogress(bag)\n# Starting operations (This will take a while)\nbag.compute()","metadata":{"execution":{"iopub.status.busy":"2024-05-20T02:12:43.737360Z","iopub.execute_input":"2024-05-20T02:12:43.737798Z","iopub.status.idle":"2024-05-20T04:26:44.935566Z","shell.execute_reply.started":"2024-05-20T02:12:43.737757Z","shell.execute_reply":"2024-05-20T04:26:44.933134Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note that the first bag ran into a ValueError with some partitions, complaining with one or more columns identified as Booleans in the reference partition used to build the features (partition5) while actually containing null values in those partitions. Function \"read_and_init_ww\" was modified to tackle this problem for the second run**\\\nWhat's great with dask Client API is that (as long as our tasks are independent) an error occuring in one task doesn't impact the others. Indeed Dask continues to execute the other tasks in the background so we don't have to rerun the first bag.","metadata":{}},{"cell_type":"code","source":"%ls feature_matrices","metadata":{"execution":{"iopub.status.busy":"2024-05-20T11:37:30.315495Z","iopub.execute_input":"2024-05-20T11:37:30.317043Z","iopub.status.idle":"2024-05-20T11:37:31.446779Z","shell.execute_reply.started":"2024-05-20T11:37:30.316956Z","shell.execute_reply":"2024-05-20T11:37:31.445223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(glob.glob(\"feature_matrices/*\")) # The number of feature matrices created should be equal to the number of partitions = 128","metadata":{"execution":{"iopub.status.busy":"2024-05-20T16:01:08.686177Z","iopub.execute_input":"2024-05-20T16:01:08.686672Z","iopub.status.idle":"2024-05-20T16:01:08.697108Z","shell.execute_reply.started":"2024-05-20T16:01:08.686637Z","shell.execute_reply":"2024-05-20T16:01:08.696226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Zip the directory to finally export it to kaggle dataset\n!zip -r fms.zip feature_matrices","metadata":{"execution":{"iopub.status.busy":"2024-05-20T16:09:46.742529Z","iopub.execute_input":"2024-05-20T16:09:46.742964Z","iopub.status.idle":"2024-05-20T16:13:02.639186Z","shell.execute_reply.started":"2024-05-20T16:09:46.742932Z","shell.execute_reply":"2024-05-20T16:13:02.637432Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import FileLink\nFileLink('fms.zip')","metadata":{"execution":{"iopub.status.busy":"2024-05-20T16:13:46.272083Z","iopub.execute_input":"2024-05-20T16:13:46.272617Z","iopub.status.idle":"2024-05-20T16:13:46.283224Z","shell.execute_reply.started":"2024-05-20T16:13:46.272579Z","shell.execute_reply":"2024-05-20T16:13:46.281019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Conclusion***\n\nIt's my first time to take more time to execute a code than to write it (funny). I hope this notebook and the associated dataset can help the competitors (even though the competition is almost over). I made the feature matrices a [dataset](http://www.kaggle.com/datasets/diarray/deep-feature-synthesis-home-credit-stability) so competitors who want to use it for the competition don't have to lose 20 hours or more running this notebook.\n\nAs you saw earlier DFS has generated 2526 features determined by the primitives we gave to the function and the logical types of the columns in the different tables, extracting good features from raw data is a task that requires intuition and experience. Deep Feature Synthesis is based on that intuition so it can generate a lot of features that a Data engineer would think of (and also a lot that we as humans would never think of) but obviously all those are not good features, getting 2000+ meaningful features just by calling a function is more about magic than computer science. This means that the features generated by DFS need to go through a thorough **feature selection** process to select the best features for training. However this task sounds much simpler than feature engineering.\n\nAdditionally, even with Dask this code may not be adapted for feature engineering in a submission notebook, we could consider several partitioning and threading options to try to decrease the time needed to calculate feature matrices for the whole dataset but we may still end up with more than a 12 hours runtime on kaggle. If the hidden test set for the competition contains indeed 90% of the numbers of case_id values of the train set, then this code may not be usable for submissions (even if we reduce the number of features to calculate).\n\n**Thanks for reading, and good luck to all the competitors for this final week**","metadata":{}},{"cell_type":"markdown","source":"## ***References***\n1. Kanter, J. M., & Veeramachaneni, K. (2015). Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA) (pp. 1-10). IEEE.\n\n2. Alteryx. (2018). Predict Loan Repayment: Automated Loan Repayment. GitHub. https://github.com/alteryx/predict-loan-repayment/blob/master/Automated%20Loan%20Repayment.ipynb\n\n3. Alteryx. (2018). Automated-Manual-Comparison: Loan Repayment. GitHub. https://github.com/alteryx/Automated-Manual-Comparison/blob/master/Loan%20Repayment/notebooks/Featuretools%20on%20Dask.ipynb\n\n4. Featuretools/Alteryx. (n.d.). Transition to Featuretools v1.0 — Featuretools 1.30.0 documentation. Retrieved [May 2024], from https://featuretools.alteryx.com/en/v1.30.0/resources/transition_to_ft_v1.0.html\n\n5. Dask Development Team (2016). Dask: Library for dynamic task scheduling\nURL http://dask.pydata.org\n","metadata":{}}]}