{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"},{"sourceId":8469662,"sourceType":"datasetVersion","datasetId":5050101}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## ***Intro***\n\nAfter the previous feature engineering step we need to select among the 2526 features a more reasonable number of features in order for the models to be as simple as possible, try to avoid overfitting and get ride of irrelevent features that can slow down learning. We may also drop some noisy training examples (if any) and implement a solution to handle null and nan values.\n\nThis notebook can be considered as the direct continuation of the [automatic feature engineering notebook](https://www.kaggle.com/code/diarray/autofeatureengineering-dfs). Herein we will drop columns with over 70% null values, columns with variance below 0.2, and columns highly correlated with other features (correlation factor above 0.8) to get ride of the least informative features. Subsequently, we will select the top 100 features from the remaining based on Information Gain estimation using K-Nearest Neighbors distance.","metadata":{}},{"cell_type":"markdown","source":"## ***Imports***","metadata":{}},{"cell_type":"code","source":"# For data manipulation and linear algebra\nimport pandas as pd\nimport numpy as np\n\n# for file manipulation\nimport glob\n# for Garbage collecting\nimport gc\n# To filter warnings\nimport warnings\n# for serialization\nimport pickle\n# To tune the default parameter of mutual_info_classif\nfrom functools import partial\n\n# For feature selection\nfrom sklearn.feature_selection import SelectKBest, mutual_info_classif, VarianceThreshold\nfrom sklearn.preprocessing import LabelEncoder","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:46:57.175646Z","iopub.execute_input":"2024-05-22T17:46:57.176676Z","iopub.status.idle":"2024-05-22T17:46:59.724063Z","shell.execute_reply.started":"2024-05-22T17:46:57.176638Z","shell.execute_reply":"2024-05-22T17:46:59.722683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Functions***","metadata":{}},{"cell_type":"code","source":"def concatenate_fms(fm_paths, columns=None):\n    \"\"\"\n    Concatenate the feature matrices, to avoid out of memory error intermediate dfs are deleted and garbage collected as we advance\n    \n    Parameters:\n    fm_paths (list of str): List of file paths to the parquet files.\n    columns (pd.Index, list[str]): The specific columns to read from the parquet file\n    \n    Returns:\n    pd.DataFrame: The concatenated Feature matrix.\n    \"\"\"\n    feature_matrix = pd.read_parquet(fm_paths[0], columns=columns)\n    for fm in fm_paths[1:]:\n        intermediate = pd.read_parquet(fm, columns=columns)\n        feature_matrix = pd.concat([feature_matrix, intermediate])\n        del intermediate\n        gc.collect()\n    return feature_matrix\n\ndef drop_toomany_nan(X_df, threshold=0.7):\n    \"\"\"\n    Drops columns from a DataFrame that have more than a specified threshold of missing values.\n    \n    Args:\n        df (pandas.DataFrame): The DataFrame to process.\n        threshold (float, optional): The percentage of missing values above which a column is dropped. Defaults to 0.7 (70%).\n    \n    Returns:\n        pandas.DataFrame: The DataFrame with columns exceeding the threshold of missing values dropped.\n    \"\"\"\n    # Calculate the percentage of missing values per column\n    missing_percentages = X_df.isnull().mean()\n    # Drop columns exceeding the threshold\n    columns_to_drop = missing_percentages[missing_percentages >= threshold].index\n    X_df.drop(columns=columns_to_drop, inplace=True)\n    return X_df\n\ndef fillna(X_df):\n    \"\"\"\n    Fills missing values in a DataFrame with specified placeholders.\n    \n    Numerical columns are filled with -9999 and non-numerical columns \n    are filled with \"<N/A>\".\n\n    Args:\n        X_df (pandas.DataFrame): The DataFrame with potential missing values.\n    \n    Returns:\n        pandas.DataFrame: The DataFrame with missing values filled.\n    \"\"\"\n    # Replace the remaining NAN values with a placeholder\n    num_placeholder = -9999\n    others_placeholder = \"<N/A>\"\n    non_num_cols = set(X_df.select_dtypes(exclude=\"number\").columns)\n    num_cols = set(X_df.columns) - non_num_cols\n    \n    # Add a category for missing values\n    for col in X_df.select_dtypes(include='category').columns:\n        X_df[col] = X_df[col].cat.add_categories(\"<N/A>\")\n        \n    X_df[list(num_cols)] = X_df[list(num_cols)].fillna(num_placeholder)\n    X_df[list(non_num_cols)] = X_df[list(non_num_cols)].fillna(others_placeholder)\n    return X_df\n\ndef to_num(X_df, label_encoders=None):\n    \"\"\"\n    Encodes categorical and boolean features in a pandas DataFrame to numerical representations.\n    This function performs two main tasks:\n    \n    1. **Converts boolean columns to integers:** \n    2. **Encodes categorical features using label encoding:** \n    \n    Args:\n        X_df (pandas.DataFrame): The DataFrame containing the features to be converted.\n        label_encoders (dict, optional): A dictionary where the keys are the column names and the values are labelEncoder instances used \n          to encode the respective columns, If None is provided the function create this dictionary and returns it.\n    \n    Returns:\n        pandas.DataFrame: The DataFrame with encoded features (if label_encoders is not None).\n        tuple[pandas.DataFrame, dict]: A tuple containing the DataFrame with encoded features and a dictionary of fitted LabelEncoders (if label_encoders is None).\n  \"\"\"\n    # Step 1: Convert boolean values to their integer representation\n    bool_cols = X_df.select_dtypes(include=\"bool\").columns\n    X_df[bool_cols] =  X_df[bool_cols].astype(int)\n\n    # Step 2: Convert categorical values to their integer representation using LabelEncoder\n    if label_encoders:\n        for col, le in label_encoders.items():\n            if col in X_df.columns:\n                # It is possible that some categories was missing in the subset used during feature selection\n                # This block will handle such cases without having to re-fit the labelEncoder\n                try:\n                    X_df[col] = le.transform(X_df[col])\n                except ValueError:\n                    # Handling new categories by updating the LabelEncoder's classes_ attribute\n                    existing_classes = set(le.classes_)\n                    new_classes = set(X_df[col].unique())\n                    combined_classes = np.array(list(existing_classes | new_classes))\n                    le.classes_ = combined_classes\n                    X_df[col] = le.transform(X_df[col])\n        return X_df\n    \n    label_encoders = {}  # To keep track of label encoders for each categorical column\n    for col in X_df.select_dtypes(include=['object', \"category\"]).columns:\n        le = LabelEncoder()\n        X_df[col] = le.fit_transform(X_df[col])\n        label_encoders[col] = le\n    return X_df, label_encoders\n\ndef drop_low_variance(X_df, threshold=0.2):\n    \"\"\"\n    Drops features from a pandas DataFrame with low variance (below a threshold).\n    \n    Args:\n        df (pandas.DataFrame): The DataFrame containing the features.\n        threshold (float, optional): The minimum variance threshold to keep a feature. Features with variance below this threshold are removed. Defaults to 0.2.\n    \n    Returns:\n      pandas.DataFrame: The DataFrame with low-variance features removed.\n    \"\"\"\n    selector = VarianceThreshold(threshold=threshold)\n    selector.fit(X_df)\n    # Get the boolean mask of features (the selected features will be indexed by one and the remainings by 0)\n    support_mask = selector.get_support()\n    # Use the mask to get the names of the remaining columns\n    selected_columns = X_df.columns[support_mask]\n    return X_df[selected_columns]\n\ndef drop_redundant_features(X_df, threshold=0.8):\n    \"\"\"\n    Drops features from a DataFrame that are highly correlated (above the threshold)\n    with another feature, keeping only one of the correlated features.\n    \n    Args:\n        df (pandas.DataFrame): The DataFrame containing the features.\n        threshold (float, optional): The correlation threshold above which features are considered highly correlated.\n    \n    Returns:\n        pandas.DataFrame: The DataFrame with redundant features removed.\n    \"\"\"\n    corr_matrix = X_df.corr()\n    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))\n    to_drop = [col for col in upper.columns if any(upper[col].abs() > threshold)]\n    X_df.drop(columns=to_drop, inplace=True)\n    return X_df\n\ndef selectKbestbyIG(X_df, target, k=100, n_neighbors=5):\n    \"\"\"\n    Selects the top K features from a pandas DataFrame for classification using Information Gain.\n    \n    Args:\n        df (pandas.DataFrame): The DataFrame containing the features.\n        target (pandas.Series): The target variable column.\n        k (int, optional): The number of features to select top K features based on information gain. (Defaults to 100.)\n    \n    Returns:\n    pandas.DataFrame: The DataFrame containing the top K features selected using mutual information.\n    \"\"\"\n    # Create a partial function to set the number of neighbors\n    mutual_info_func = partial(mutual_info_classif, n_neighbors=n_neighbors)\n    \n    selector = SelectKBest(mutual_info_func, k=k)\n    selector.fit(X_df, target)\n    # Get the boolean mask of features (the selected features will be indexed by one and the remainings by 0)\n    support_mask = selector.get_support()\n    # Use the mask to get the names of the remaining columns\n    selected_columns = X_df.columns[support_mask]\n    return X_df[selected_columns]","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:47:01.047899Z","iopub.execute_input":"2024-05-22T17:47:01.048935Z","iopub.status.idle":"2024-05-22T17:47:01.171252Z","shell.execute_reply.started":"2024-05-22T17:47:01.048895Z","shell.execute_reply":"2024-05-22T17:47:01.170364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Feature Selection***\n\n**Use a subset of 32 features matrices (25% of the dataset) to perform feature selection. The selected features will then be generalized to the entire dataset.**","metadata":{}},{"cell_type":"code","source":"# Paths to the feature matrices\nfms_paths = glob.glob(\"/kaggle/input/deep-feature-synthesis-home-credit-stability/feature_matrices/*\")\n# We will do the feature selection with a randomly shuffled subset (approximately 25% of the entire dataset)\nfms_paths[:10]","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:40:51.733565Z","iopub.execute_input":"2024-05-22T12:40:51.733990Z","iopub.status.idle":"2024-05-22T12:40:51.743610Z","shell.execute_reply.started":"2024-05-22T12:40:51.733952Z","shell.execute_reply":"2024-05-22T12:40:51.742627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:41:41.381447Z","iopub.execute_input":"2024-05-22T12:41:41.381912Z","iopub.status.idle":"2024-05-22T12:41:41.386934Z","shell.execute_reply.started":"2024-05-22T12:41:41.381872Z","shell.execute_reply":"2024-05-22T12:41:41.385795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset = concatenate_fms(fms_paths[:32])","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:42:35.308196Z","iopub.execute_input":"2024-05-22T12:42:35.308931Z","iopub.status.idle":"2024-05-22T12:43:53.799490Z","shell.execute_reply.started":"2024-05-22T12:42:35.308893Z","shell.execute_reply":"2024-05-22T12:43:53.798321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset.shape","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:44:40.397218Z","iopub.execute_input":"2024-05-22T12:44:40.398064Z","iopub.status.idle":"2024-05-22T12:44:40.404461Z","shell.execute_reply.started":"2024-05-22T12:44:40.398031Z","shell.execute_reply":"2024-05-22T12:44:40.403414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Function \"drop_toomany_nan\" has been edited, its previous version (used for feature selction) emcompassed function \"fillna\"\nsubset = drop_toomany_nan(X_df=subset)","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:44:45.816463Z","iopub.execute_input":"2024-05-22T12:44:45.817100Z","iopub.status.idle":"2024-05-22T12:45:02.642563Z","shell.execute_reply.started":"2024-05-22T12:44:45.817065Z","shell.execute_reply":"2024-05-22T12:45:02.641600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:45:02.644200Z","iopub.execute_input":"2024-05-22T12:45:02.644515Z","iopub.status.idle":"2024-05-22T12:45:04.668034Z","shell.execute_reply.started":"2024-05-22T12:45:02.644490Z","shell.execute_reply":"2024-05-22T12:45:04.667066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Encode non numeric features\nsubset, lbl_encorders = to_num(subset)","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:45:12.158794Z","iopub.execute_input":"2024-05-22T12:45:12.159773Z","iopub.status.idle":"2024-05-22T12:45:18.796611Z","shell.execute_reply.started":"2024-05-22T12:45:12.159738Z","shell.execute_reply":"2024-05-22T12:45:18.795696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:45:18.798192Z","iopub.execute_input":"2024-05-22T12:45:18.798488Z","iopub.status.idle":"2024-05-22T12:45:18.951322Z","shell.execute_reply.started":"2024-05-22T12:45:18.798465Z","shell.execute_reply":"2024-05-22T12:45:18.950337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop features with low variance\nsubset = drop_low_variance(subset)","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:48:28.596933Z","iopub.execute_input":"2024-05-22T12:48:28.597362Z","iopub.status.idle":"2024-05-22T12:48:43.331047Z","shell.execute_reply.started":"2024-05-22T12:48:28.597331Z","shell.execute_reply":"2024-05-22T12:48:43.330157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subset","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:48:55.749339Z","iopub.execute_input":"2024-05-22T12:48:55.750476Z","iopub.status.idle":"2024-05-22T12:48:56.053205Z","shell.execute_reply.started":"2024-05-22T12:48:55.750437Z","shell.execute_reply":"2024-05-22T12:48:56.052196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr_matrix = subset.corr()\ncorr_matrix","metadata":{"execution":{"iopub.status.busy":"2024-05-22T12:51:15.627210Z","iopub.execute_input":"2024-05-22T12:51:15.628051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))\nupper","metadata":{"execution":{"iopub.status.busy":"2024-05-22T13:23:26.270620Z","iopub.execute_input":"2024-05-22T13:23:26.271113Z","iopub.status.idle":"2024-05-22T13:23:26.328907Z","shell.execute_reply.started":"2024-05-22T13:23:26.271079Z","shell.execute_reply":"2024-05-22T13:23:26.327853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"to_drop = [col for col in upper.columns if any(upper[col].abs() >= 0.8)]","metadata":{"execution":{"iopub.status.busy":"2024-05-22T13:24:57.370067Z","iopub.execute_input":"2024-05-22T13:24:57.370987Z","iopub.status.idle":"2024-05-22T13:24:57.997768Z","shell.execute_reply.started":"2024-05-22T13:24:57.370953Z","shell.execute_reply":"2024-05-22T13:24:57.996607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(to_drop))\n# Drop redundant features\nsubset.drop(columns=to_drop, inplace=True)\nsubset","metadata":{"execution":{"iopub.status.busy":"2024-05-22T13:27:18.949886Z","iopub.execute_input":"2024-05-22T13:27:18.950885Z","iopub.status.idle":"2024-05-22T13:27:21.643772Z","shell.execute_reply.started":"2024-05-22T13:27:18.950832Z","shell.execute_reply":"2024-05-22T13:27:21.642580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = pd.read_parquet(\"/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_base.parquet\", \n                         columns=[\"case_id\", \"target\"])\ntarget","metadata":{"execution":{"iopub.status.busy":"2024-05-22T13:52:04.242793Z","iopub.execute_input":"2024-05-22T13:52:04.243515Z","iopub.status.idle":"2024-05-22T13:52:04.392413Z","shell.execute_reply.started":"2024-05-22T13:52:04.243481Z","shell.execute_reply":"2024-05-22T13:52:04.391403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = target[target[\"case_id\"].isin(subset.index)].set_index(\"case_id\")\ntarget","metadata":{"execution":{"iopub.status.busy":"2024-05-22T14:12:55.104836Z","iopub.execute_input":"2024-05-22T14:12:55.105722Z","iopub.status.idle":"2024-05-22T14:12:55.172014Z","shell.execute_reply.started":"2024-05-22T14:12:55.105685Z","shell.execute_reply":"2024-05-22T14:12:55.170950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = target.reindex(subset.index)[\"target\"]\ntarget","metadata":{"execution":{"iopub.status.busy":"2024-05-22T14:19:03.720184Z","iopub.execute_input":"2024-05-22T14:19:03.721206Z","iopub.status.idle":"2024-05-22T14:19:03.747025Z","shell.execute_reply.started":"2024-05-22T14:19:03.721167Z","shell.execute_reply":"2024-05-22T14:19:03.745977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Select the 100 most infomative features (according to Information Gain)\nsubset = selectKbestbyIG(X_df=subset, target=target)\nsubset","metadata":{"execution":{"iopub.status.busy":"2024-05-22T14:21:41.426141Z","iopub.execute_input":"2024-05-22T14:21:41.427166Z","iopub.status.idle":"2024-05-22T14:35:11.871220Z","shell.execute_reply.started":"2024-05-22T14:21:41.427127Z","shell.execute_reply":"2024-05-22T14:35:11.869990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Save intermediate objects***\n\n**Save the subset dataframe with the 100 selected features to parquet and the label encoders with pickle. Those will be used to generalize the results to the rest of original the features matrices**","metadata":{}},{"cell_type":"code","source":"# Save the reference Dataframe \nsubset.to_parquet(\"feature_selection.parquet\")\n%ls","metadata":{"execution":{"iopub.status.busy":"2024-05-22T15:15:55.027309Z","iopub.execute_input":"2024-05-22T15:15:55.027720Z","iopub.status.idle":"2024-05-22T15:15:57.720136Z","shell.execute_reply.started":"2024-05-22T15:15:55.027690Z","shell.execute_reply":"2024-05-22T15:15:57.718891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\nwith open('labelEncoders.pkl', 'wb') as file:\n    # Pickle the label encoders dictionary using the highest protocol available.\n    pickle.dump(lbl_encorders, file, pickle.HIGHEST_PROTOCOL)","metadata":{"execution":{"iopub.status.busy":"2024-05-22T15:44:04.565593Z","iopub.execute_input":"2024-05-22T15:44:04.566532Z","iopub.status.idle":"2024-05-22T15:44:04.604457Z","shell.execute_reply.started":"2024-05-22T15:44:04.566498Z","shell.execute_reply":"2024-05-22T15:44:04.603538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Create the final training dataset***\n\n**Create a base table with the 100 selected features and the target holding the data for all the training examples (case_ids)**","metadata":{}},{"cell_type":"code","source":"# Paths to the feature matrices (ordered this time)\nfms_paths = glob.glob(\"/kaggle/input/deep-feature-synthesis-home-credit-stability/feature_matrices/*\")\nfms_paths.sort(key=lambda fm_path: int(fm_path.split(\"_\")[1][18:]))\nfms_paths[:10]","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:47:09.315349Z","iopub.execute_input":"2024-05-22T17:47:09.315942Z","iopub.status.idle":"2024-05-22T17:47:09.348901Z","shell.execute_reply.started":"2024-05-22T17:47:09.315911Z","shell.execute_reply":"2024-05-22T17:47:09.348064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open('labelEncoders.pkl', 'rb') as file:\n    lbl_encoders = pickle.load(file)","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:47:12.821908Z","iopub.execute_input":"2024-05-22T17:47:12.822582Z","iopub.status.idle":"2024-05-22T17:47:12.840511Z","shell.execute_reply.started":"2024-05-22T17:47:12.822538Z","shell.execute_reply":"2024-05-22T17:47:12.839571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selected100features = list(pd.read_parquet(\"feature_selection.parquet\").columns)\n# This is because WEEK_NUM has not been selected as one of the 100 most important features,...\n# but we need it for the stability metric (competition context)\nselected100features.insert(0, \"WEEK_NUM\")\nselected100features[:10]","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:48:50.529440Z","iopub.execute_input":"2024-05-22T17:48:50.530520Z","iopub.status.idle":"2024-05-22T17:48:50.919889Z","shell.execute_reply.started":"2024-05-22T17:48:50.530463Z","shell.execute_reply":"2024-05-22T17:48:50.918502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = concatenate_fms(fms_paths, columns=selected100features)\ndataset = fillna(dataset)\ndataset = to_num(X_df=dataset, label_encoders=lbl_encoders)\ndataset.head()","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:49:25.156103Z","iopub.execute_input":"2024-05-22T17:49:25.156563Z","iopub.status.idle":"2024-05-22T17:51:57.747364Z","shell.execute_reply.started":"2024-05-22T17:49:25.156513Z","shell.execute_reply":"2024-05-22T17:51:57.746209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset.shape","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:52:22.437385Z","iopub.execute_input":"2024-05-22T17:52:22.437856Z","iopub.status.idle":"2024-05-22T17:52:22.446105Z","shell.execute_reply.started":"2024-05-22T17:52:22.437820Z","shell.execute_reply":"2024-05-22T17:52:22.444762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = pd.read_parquet(\"/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_base.parquet\", \n                         columns=[\"case_id\", \"target\"]).set_index(\"case_id\")[\"target\"]\ntarget","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:58:27.771054Z","iopub.execute_input":"2024-05-22T17:58:27.771461Z","iopub.status.idle":"2024-05-22T17:58:27.854187Z","shell.execute_reply.started":"2024-05-22T17:58:27.771430Z","shell.execute_reply":"2024-05-22T17:58:27.852959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create the final base table with the selected features and the target\ndataset[\"target\"] = target\ndataset","metadata":{"execution":{"iopub.status.busy":"2024-05-22T17:59:39.597657Z","iopub.execute_input":"2024-05-22T17:59:39.598055Z","iopub.status.idle":"2024-05-22T17:59:40.381872Z","shell.execute_reply.started":"2024-05-22T17:59:39.598024Z","shell.execute_reply":"2024-05-22T17:59:40.379416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save the final table to csv and parquet\ndataset.to_parquet(\"base_100features.parquet\")\ndataset.to_csv(\"base_100features.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-05-22T18:03:55.389118Z","iopub.execute_input":"2024-05-22T18:03:55.390759Z","iopub.status.idle":"2024-05-22T18:06:27.702174Z","shell.execute_reply.started":"2024-05-22T18:03:55.390685Z","shell.execute_reply":"2024-05-22T18:06:27.700724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%ls","metadata":{"execution":{"iopub.status.busy":"2024-05-22T18:06:35.493916Z","iopub.execute_input":"2024-05-22T18:06:35.494358Z","iopub.status.idle":"2024-05-22T18:06:36.661758Z","shell.execute_reply.started":"2024-05-22T18:06:35.494323Z","shell.execute_reply":"2024-05-22T18:06:36.660193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ***Conclusion***\n\nFollowing our feature engineering steps, we needed to narrow down the 2526 features to a more manageable number to simplify the model, minimize overfitting, and eliminate irrelevant features that could hinder learning. This operation was much less time consuming than the feature engineering process, but note that the different steps in this notebook are highly customizable and this is just a baseline.\n\n**Thanks for reading**","metadata":{}}]}