{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30699,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import gc\nimport lightgbm as lgb  # type: ignore\nimport numpy as np  # type: ignore\nimport pandas as pd  # type: ignore\nimport polars as pl  # type: ignore\nimport warnings\n\nfrom catboost import CatBoostClassifier, Pool  # type: ignore\nfrom glob import glob\nfrom IPython.display import display  # type: ignore\nfrom pathlib import Path\nfrom sklearn.base import BaseEstimator, ClassifierMixin  # type: ignore\nfrom sklearn.metrics import roc_auc_score  # type: ignore\nfrom sklearn.model_selection import StratifiedGroupKFold  # type: ignore\nfrom typing import Any\n\nwarnings.filterwarnings(\"ignore\")\n\nROOT = Path(\"/kaggle/input/home-credit-credit-risk-model-stability\")\nTRAIN_DIR = ROOT / \"parquet_files\" / \"train\"\nTEST_DIR = ROOT / \"parquet_files\" / \"test\"","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:15.843015Z","iopub.execute_input":"2024-05-14T11:00:15.843882Z","iopub.status.idle":"2024-05-14T11:00:21.892297Z","shell.execute_reply.started":"2024-05-14T11:00:15.843848Z","shell.execute_reply":"2024-05-14T11:00:21.891508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Utility:\n    @staticmethod\n    def get_feat_defs(ending_with: str) -> None:\n        \"\"\"\n        Retrieves feature definitions from a CSV file based on the specified ending.\n\n        Args:\n        - ending_with (str): Ending to filter feature definitions.\n\n        Returns:\n        - pl.DataFrame: Filtered feature definitions.\n        \"\"\"\n        feat_defs: pl.DataFrame = pl.read_csv(ROOT / \"feature_definitions.csv\")\n\n        filtered_feats: pl.DataFrame = feat_defs.filter(\n            pl.col(\"Variable\").apply(lambda var: var.endswith(ending_with))\n        )\n\n        with pl.Config(fmt_str_lengths=200, tbl_rows=-1):\n            print(filtered_feats)\n\n        filtered_feats = None\n        feat_defs = None\n\n    @staticmethod\n    def find_index(lst: list[Any], item: Any) -> int | None:\n        \"\"\"\n        Finds the index of an item in a list.\n\n        Args:\n        - lst (list): List to search.\n        - item (Any): Item to find in the list.\n\n        Returns:\n        - int | None: Index of the item if found, otherwise None.\n        \"\"\"\n        try:\n            return lst.index(item)\n        except ValueError:\n            return None\n\n    @staticmethod\n    def dtype_to_str(dtype: pl.DataType) -> str:\n        \"\"\"\n        Converts Polars data type to string representation.\n\n        Args:\n        - dtype (pl.DataType): Polars data type.\n\n        Returns:\n        - str: String representation of the data type.\n        \"\"\"\n        dtype_map = {\n            pl.Decimal: \"Decimal\",\n            pl.Float32: \"Float32\",\n            pl.Float64: \"Float64\",\n            pl.UInt8: \"UInt8\",\n            pl.UInt16: \"UInt16\",\n            pl.UInt32: \"UInt32\",\n            pl.UInt64: \"UInt64\",\n            pl.Int8: \"Int8\",\n            pl.Int16: \"Int16\",\n            pl.Int32: \"Int32\",\n            pl.Int64: \"Int64\",\n            pl.Date: \"Date\",\n            pl.Datetime: \"Datetime\",\n            pl.Duration: \"Duration\",\n            pl.Time: \"Time\",\n            pl.Array: \"Array\",\n            pl.List: \"List\",\n            pl.Struct: \"Struct\",\n            pl.String: \"String\",\n            pl.Categorical: \"Categorical\",\n            pl.Enum: \"Enum\",\n            pl.Utf8: \"Utf8\",\n            pl.Binary: \"Binary\",\n            pl.Boolean: \"Boolean\",\n            pl.Null: \"Null\",\n            pl.Object: \"Object\",\n            pl.Unknown: \"Unknown\",\n        }\n\n        return dtype_map.get(dtype)\n\n    @staticmethod\n    def find_feat_occur(regex_path: str, ending_with: str) -> pl.DataFrame:\n        \"\"\"\n        Finds occurrences of features ending with a specific string in Parquet files.\n\n        Args:\n        - regex_path (str): Regular expression to match Parquet file paths.\n        - ending_with (str): Ending to filter feature names.\n\n        Returns:\n        - pl.DataFrame: DataFrame containing feature definitions, data types, and file locations.\n        \"\"\"\n        feat_defs: pl.DataFrame = pl.read_csv(ROOT / \"feature_definitions.csv\").filter(\n            pl.col(\"Variable\").apply(lambda var: var.endswith(ending_with))\n        )\n        feat_defs.sort(by=[\"Variable\"])\n\n        feats: list[pl.String] = feat_defs[\"Variable\"].to_list()\n        feats.sort()\n\n        occurrences: list[list] = [[set(), set()] for _ in range(feat_defs.height)]\n\n        for path in glob(str(regex_path)):\n            df_schema: dict = pl.read_parquet_schema(path)\n\n            for feat, dtype in df_schema.items():\n                index: int = Utility.find_index(feats, feat)\n                if index != None:\n                    occurrences[index][0].add(Utility.dtype_to_str(dtype))\n                    occurrences[index][1].add(Path(path).stem)\n\n        data_types: list[str] = [None] * feat_defs.height\n        file_locs: list[str] = [None] * feat_defs.height\n\n        for i, feat in enumerate(feats):\n            data_types[i] = list(occurrences[i][0])\n            file_locs[i] = list(occurrences[i][1])\n\n        feat_defs = feat_defs.with_columns(pl.Series(data_types).alias(\"Data_Type(s)\"))\n        feat_defs = feat_defs.with_columns(pl.Series(file_locs).alias(\"File_Loc(s)\"))\n\n        return feat_defs\n\n    def reduce_memory_usage(df: pl.DataFrame, name) -> pl.DataFrame:\n        \"\"\"\n        Reduces memory usage of a DataFrame by converting column types.\n\n        Args:\n        - df (pl.DataFrame): DataFrame to optimize.\n        - name (str): Name of the DataFrame.\n\n        Returns:\n        - pl.DataFrame: Optimized DataFrame.\n        \"\"\"\n        print(\n            f\"Memory usage of dataframe \\\"{name}\\\" is {round(df.estimated_size('mb'), 4)} MB.\"\n        )\n\n        int_types = [\n            pl.Int8,\n            pl.Int16,\n            pl.Int32,\n            pl.Int64,\n            pl.UInt8,\n            pl.UInt16,\n            pl.UInt32,\n            pl.UInt64,\n        ]\n        float_types = [pl.Float32, pl.Float64]\n\n        for col in df.columns:\n            col_type = df[col].dtype\n            if col_type in int_types + float_types:\n                c_min = df[col].min()\n                c_max = df[col].max()\n\n                if c_min is not None and c_max is not None:\n                    if col_type in int_types:\n                        if c_min >= 0:\n                            if (\n                                c_min >= np.iinfo(np.uint8).min\n                                and c_max <= np.iinfo(np.uint8).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.UInt8))\n                            elif (\n                                c_min >= np.iinfo(np.uint16).min\n                                and c_max <= np.iinfo(np.uint16).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.UInt16))\n                            elif (\n                                c_min >= np.iinfo(np.uint32).min\n                                and c_max <= np.iinfo(np.uint32).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.UInt32))\n                            elif (\n                                c_min >= np.iinfo(np.uint64).min\n                                and c_max <= np.iinfo(np.uint64).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.UInt64))\n                        else:\n                            if (\n                                c_min >= np.iinfo(np.int8).min\n                                and c_max <= np.iinfo(np.int8).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.Int8))\n                            elif (\n                                c_min >= np.iinfo(np.int16).min\n                                and c_max <= np.iinfo(np.int16).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.Int16))\n                            elif (\n                                c_min >= np.iinfo(np.int32).min\n                                and c_max <= np.iinfo(np.int32).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.Int32))\n                            elif (\n                                c_min >= np.iinfo(np.int64).min\n                                and c_max <= np.iinfo(np.int64).max\n                            ):\n                                df = df.with_columns(df[col].cast(pl.Int64))\n                    elif col_type in float_types:\n                        if (\n                            c_min > np.finfo(np.float32).min\n                            and c_max < np.finfo(np.float32).max\n                        ):\n                            df = df.with_columns(df[col].cast(pl.Float32))\n\n        print(\n            f\"Memory usage of dataframe \\\"{name}\\\" became {round(df.estimated_size('mb'), 4)} MB.\"\n        )\n\n        return df\n\n    def to_pandas(df: pl.DataFrame, cat_cols: list[str] = None) -> (pd.DataFrame, list[str]):  # type: ignore\n        \"\"\"\n        Converts a Polars DataFrame to a Pandas DataFrame.\n\n        Args:\n        - df (pl.DataFrame): Polars DataFrame to convert.\n        - cat_cols (list[str]): List of categorical columns. Default is None.\n\n        Returns:\n        - (pd.DataFrame, list[str]): Tuple containing the converted Pandas DataFrame and categorical columns.\n        \"\"\"\n        df: pd.DataFrame = df.to_pandas()\n\n        if cat_cols is None:\n            cat_cols = list(df.select_dtypes(\"object\").columns)\n\n        df[cat_cols] = df[cat_cols].astype(\"str\")\n\n        return df, cat_cols","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:21.894385Z","iopub.execute_input":"2024-05-14T11:00:21.894968Z","iopub.status.idle":"2024-05-14T11:00:21.929135Z","shell.execute_reply.started":"2024-05-14T11:00:21.894934Z","shell.execute_reply":"2024-05-14T11:00:21.928081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# feat_defs:pl.DataFrame = Utility.find_feat_occur(TRAIN_DIR / \"train_*.parquet\", \"P\")\n# feat_defs:pl.DataFrame = Utility.find_feat_occur(TRAIN_DIR / \"train_*.parquet\", \"M\")\n# feat_defs:pl.DataFrame = Utility.find_feat_occur(TRAIN_DIR / \"train_*.parquet\", \"A\")\n# feat_defs:pl.DataFrame = Utility.find_feat_occur(TRAIN_DIR / \"train_*.parquet\", \"D\")\n# feat_defs:pl.DataFrame = Utility.find_feat_occur(TRAIN_DIR / \"train_*.parquet\", \"T\")\n# feat_defs:pl.DataFrame = Utility.find_feat_occur(TRAIN_DIR / \"train_*.parquet\", \"L\")\n# feat_defs:pl.DataFrame = pl.read_csv(ROOT / \"feature_definitions.csv\")\n# with pl.Config(fmt_str_lengths=1000, tbl_rows=-1, tbl_width_chars=180):\n#     print(feat_defs)","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:21.930636Z","iopub.execute_input":"2024-05-14T11:00:21.930897Z","iopub.status.idle":"2024-05-14T11:00:21.948513Z","shell.execute_reply.started":"2024-05-14T11:00:21.930875Z","shell.execute_reply":"2024-05-14T11:00:21.947821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Aggregator:\n    @staticmethod\n    def max_expr(df: pl.LazyFrame) -> list[pl.Series]:\n        \"\"\"\n        Generates expressions for calculating maximum values for specific columns.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - list[pl.Series]: List of expressions for maximum values.\n        \"\"\"\n        cols: list[str] = [\n            col\n            for col in df.columns\n            if (col[-1] in (\"P\", \"M\", \"A\", \"D\", \"T\", \"L\")) or (\"num_group\" in col)\n        ]\n\n        expr_max: list[pl.Series] = [\n            pl.col(col).max().alias(f\"max_{col}\") for col in cols\n        ]\n\n        return expr_max\n\n    @staticmethod\n    def min_expr(df: pl.LazyFrame) -> list[pl.Series]:\n        \"\"\"\n        Generates expressions for calculating minimum values for specific columns.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - list[pl.Series]: List of expressions for minimum values.\n        \"\"\"\n        cols: list[str] = [\n            col\n            for col in df.columns\n            if (col[-1] in (\"P\", \"M\", \"A\", \"D\", \"T\", \"L\")) or (\"num_group\" in col)\n        ]\n\n        expr_min: list[pl.Series] = [\n            pl.col(col).min().alias(f\"min_{col}\") for col in cols\n        ]\n\n        return expr_min\n\n    @staticmethod\n    def mean_expr(df: pl.LazyFrame) -> list[pl.Series]:\n        \"\"\"\n        Generates expressions for calculating mean values for specific columns.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - list[pl.Series]: List of expressions for mean values.\n        \"\"\"\n        cols: list[str] = [col for col in df.columns if col.endswith((\"P\", \"A\", \"D\"))]\n\n        expr_mean: list[pl.Series] = [\n            pl.col(col).mean().alias(f\"mean_{col}\") for col in cols\n        ]\n\n        return expr_mean\n\n    @staticmethod\n    def var_expr(df: pl.LazyFrame) -> list[pl.Series]:\n        \"\"\"\n        Generates expressions for calculating variance for specific columns.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - list[pl.Series]: List of expressions for variance.\n        \"\"\"\n        cols: list[str] = [col for col in df.columns if col.endswith((\"P\", \"A\", \"D\"))]\n\n        expr_mean: list[pl.Series] = [\n            pl.col(col).var().alias(f\"var_{col}\") for col in cols\n        ]\n\n        return expr_mean\n\n    @staticmethod\n    def mode_expr(df: pl.LazyFrame) -> list[pl.Series]:\n        \"\"\"\n        Generates expressions for calculating mode values for specific columns.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - list[pl.Series]: List of expressions for mode values.\n        \"\"\"\n        cols: list[str] = [col for col in df.columns if col.endswith(\"M\")]\n\n        expr_mode: list[pl.Series] = [\n            pl.col(col).drop_nulls().mode().first().alias(f\"mode_{col}\") for col in cols\n        ]\n\n        return expr_mode\n\n    @staticmethod\n    def get_exprs(df: pl.LazyFrame) -> list[pl.Series]:\n        \"\"\"\n        Combines expressions for maximum, mean, and variance calculations.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - list[pl.Series]: List of combined expressions.\n        \"\"\"\n        exprs = (\n            Aggregator.max_expr(df) + Aggregator.mean_expr(df) + Aggregator.var_expr(df)\n        )\n\n        return exprs","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:21.951231Z","iopub.execute_input":"2024-05-14T11:00:21.951616Z","iopub.status.idle":"2024-05-14T11:00:21.967391Z","shell.execute_reply.started":"2024-05-14T11:00:21.951578Z","shell.execute_reply":"2024-05-14T11:00:21.9665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class SchemaGen:\n    @staticmethod\n    def change_dtypes(df: pl.LazyFrame) -> pl.LazyFrame:\n        \"\"\"\n        Changes the data types of columns in the DataFrame.\n\n        Args:\n        - df (pl.LazyFrame): Input LazyFrame.\n\n        Returns:\n        - pl.LazyFrame: LazyFrame with modified data types.\n        \"\"\"\n        for col in df.columns:\n            if col == \"case_id\":\n                df = df.with_columns(pl.col(col).cast(pl.UInt32).alias(col))\n            elif col in [\"WEEK_NUM\", \"num_group1\", \"num_group2\"]:\n                df = df.with_columns(pl.col(col).cast(pl.UInt16).alias(col))\n            elif col == \"date_decision\" or col[-1] == \"D\":\n                df = df.with_columns(pl.col(col).cast(pl.Date).alias(col))\n            elif col[-1] in [\"P\", \"A\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Float64).alias(col))\n            elif col[-1] in (\"M\",):\n                df = df.with_columns(pl.col(col).cast(pl.String))\n        return df\n\n    @staticmethod\n    def scan_files(glob_path: str, depth: int = None) -> pl.LazyFrame:\n        \"\"\"\n        Scans Parquet files matching the glob pattern and combines them into a LazyFrame.\n\n        Args:\n        - glob_path (str): Glob pattern to match Parquet files.\n        - depth (int, optional): Depth level for data aggregation. Defaults to None.\n\n        Returns:\n        - pl.LazyFrame: Combined LazyFrame.\n        \"\"\"\n        chunks: list[pl.LazyFrame] = []\n        for path in glob(str(glob_path)):\n            df: pl.LazyFrame = pl.scan_parquet(\n                path, low_memory=True, rechunk=True\n            ).pipe(SchemaGen.change_dtypes)\n            print(f\"File {Path(path).stem} loaded into memory.\")\n\n            if depth in (1, 2):\n                exprs: list[pl.Series] = Aggregator.get_exprs(df)\n                df = df.group_by(\"case_id\").agg(exprs)\n\n                del exprs\n                gc.collect()\n\n            chunks.append(df)\n\n        df = pl.concat(chunks, how=\"vertical_relaxed\")\n\n        del chunks\n        gc.collect()\n\n        df = df.unique(subset=[\"case_id\"])\n\n        return df\n\n    @staticmethod\n    def join_dataframes(\n        df_base: pl.LazyFrame,\n        depth_0: list[pl.LazyFrame],\n        depth_1: list[pl.LazyFrame],\n        depth_2: list[pl.LazyFrame],\n    ) -> pl.DataFrame:\n        \"\"\"\n        Joins multiple LazyFrames with a base LazyFrame.\n\n        Args:\n        - df_base (pl.LazyFrame): Base LazyFrame.\n        - depth_0 (list[pl.LazyFrame]): List of LazyFrames for depth 0.\n        - depth_1 (list[pl.LazyFrame]): List of LazyFrames for depth 1.\n        - depth_2 (list[pl.LazyFrame]): List of LazyFrames for depth 2.\n\n        Returns:\n        - pl.DataFrame: Joined DataFrame.\n        \"\"\"\n        for i, df in enumerate(depth_0 + depth_1 + depth_2):\n            df_base = df_base.join(df, how=\"left\", on=\"case_id\", suffix=f\"_{i}\")\n\n        return df_base.collect().pipe(Utility.reduce_memory_usage, \"df_train\")","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:21.968527Z","iopub.execute_input":"2024-05-14T11:00:21.968825Z","iopub.status.idle":"2024-05-14T11:00:21.983761Z","shell.execute_reply.started":"2024-05-14T11:00:21.968802Z","shell.execute_reply":"2024-05-14T11:00:21.983019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def filter_cols(df: pl.DataFrame) -> pl.DataFrame:\n    \"\"\"\n    Filters columns in the DataFrame based on null percentage and unique values for string columns.\n\n    Args:\n    - df (pl.DataFrame): Input DataFrame.\n\n    Returns:\n    - pl.DataFrame: DataFrame with filtered columns.\n    \"\"\"\n    for col in df.columns:\n        if col not in [\"case_id\", \"year\", \"month\", \"week_num\", \"target\"]:\n            null_pct = df[col].is_null().mean()\n\n            if null_pct > 0.95:\n                df = df.drop(col)\n\n    for col in df.columns:\n        if (col not in [\"case_id\", \"year\", \"month\", \"week_num\", \"target\"]) & (\n            df[col].dtype == pl.String\n        ):\n            freq = df[col].n_unique()\n\n            if (freq > 200) | (freq == 1):\n                df = df.drop(col)\n\n    return df\n\n\ndef transform_cols(df: pl.DataFrame) -> pl.DataFrame:\n    \"\"\"\n    Transforms columns in the DataFrame according to predefined rules.\n\n    Args:\n    - df (pl.DataFrame): Input DataFrame.\n\n    Returns:\n    - pl.DataFrame: DataFrame with transformed columns.\n    \"\"\"\n    if \"riskassesment_302T\" in df.columns:\n        if df[\"riskassesment_302T\"].dtype == pl.Null:\n            df = df.with_columns(\n                [\n                    pl.Series(\n                        \"riskassesment_302T_rng\", df[\"riskassesment_302T\"], pl.UInt8\n                    ),\n                    pl.Series(\n                        \"riskassesment_302T_mean\", df[\"riskassesment_302T\"], pl.UInt8\n                    ),\n                ]\n            )\n        else:\n            pct_low: pl.Series = (\n                df[\"riskassesment_302T\"]\n                .str.split(\" - \")\n                .apply(lambda x: x[0].replace(\"%\", \"\"))\n                .cast(pl.UInt8)\n            )\n            pct_high: pl.Series = (\n                df[\"riskassesment_302T\"]\n                .str.split(\" - \")\n                .apply(lambda x: x[1].replace(\"%\", \"\"))\n                .cast(pl.UInt8)\n            )\n\n            diff: pl.Series = pct_high - pct_low\n            avg: pl.Series = ((pct_low + pct_high) / 2).cast(pl.Float32)\n\n            del pct_high, pct_low\n            gc.collect()\n\n            df = df.with_columns(\n                [\n                    diff.alias(\"riskassesment_302T_rng\"),\n                    avg.alias(\"riskassesment_302T_mean\"),\n                ]\n            )\n\n        df.drop(\"riskassesment_302T\")\n\n    return df\n\n\ndef handle_dates(df: pl.DataFrame) -> pl.DataFrame:\n    \"\"\"\n    Handles date columns in the DataFrame.\n\n    Args:\n    - df (pl.DataFrame): Input DataFrame.\n\n    Returns:\n    - pl.DataFrame: DataFrame with transformed date columns.\n    \"\"\"\n    for col in df.columns:\n        if col.endswith(\"D\"):\n            df = df.with_columns(pl.col(col) - pl.col(\"date_decision\"))\n            df = df.with_columns(pl.col(col).dt.total_days().cast(pl.Int32))\n\n    df = df.rename(\n        {\n            \"MONTH\": \"month\",\n            \"WEEK_NUM\": \"week_num\"\n        }\n    )\n            \n    df = df.with_columns(\n        [\n            pl.col(\"date_decision\").dt.year().alias(\"year\").cast(pl.Int16),\n            pl.col(\"date_decision\").dt.day().alias(\"day\").cast(pl.UInt8),\n        ]\n    )\n\n    return df.drop(\"date_decision\")","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:21.985005Z","iopub.execute_input":"2024-05-14T11:00:21.98543Z","iopub.status.idle":"2024-05-14T11:00:22.002501Z","shell.execute_reply.started":"2024-05-14T11:00:21.985399Z","shell.execute_reply":"2024-05-14T11:00:22.001742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_store: dict = {\n    \"df_base\": SchemaGen.scan_files(TRAIN_DIR / \"train_base.parquet\"),\n    \"depth_0\": [\n        SchemaGen.scan_files(TRAIN_DIR / \"train_static_cb_0.parquet\"),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        SchemaGen.scan_files(TRAIN_DIR / \"train_applprev_1_*.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_tax_registry_a_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_tax_registry_b_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_tax_registry_c_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_credit_bureau_a_1_*.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_other_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_person_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_deposit_1.parquet\", 1),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        SchemaGen.scan_files(TRAIN_DIR / \"train_credit_bureau_a_2_*.parquet\", 2),\n        SchemaGen.scan_files(TRAIN_DIR / \"train_credit_bureau_b_2.parquet\", 2),\n    ],\n}\n\ndf_train: pl.LazyFrame = (\n    SchemaGen.join_dataframes(**data_store)\n    .pipe(filter_cols)\n    .pipe(transform_cols)\n    .pipe(handle_dates)\n    .pipe(Utility.reduce_memory_usage, \"df_train\")\n)\n\ndel data_store\ngc.collect()\n\nprint(f\"Train data shape: {df_train.shape}\")\ndisplay(df_train.head(10))\n\n# df_train.write_parquet(\"train_final.parquet\", compression=\"lz4\")","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:00:22.003628Z","iopub.execute_input":"2024-05-14T11:00:22.004359Z","iopub.status.idle":"2024-05-14T11:03:28.616257Z","shell.execute_reply.started":"2024-05-14T11:00:22.004327Z","shell.execute_reply":"2024-05-14T11:03:28.615416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_store: dict = {\n    \"df_base\": SchemaGen.scan_files(TEST_DIR / \"test_base.parquet\"),\n    \"depth_0\": [\n        SchemaGen.scan_files(TEST_DIR / \"test_static_cb_0.parquet\"),\n        SchemaGen.scan_files(TEST_DIR / \"test_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        SchemaGen.scan_files(TEST_DIR / \"test_applprev_1_*.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_tax_registry_a_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_tax_registry_b_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_tax_registry_c_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_credit_bureau_a_1_*.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_credit_bureau_b_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_other_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_person_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_deposit_1.parquet\", 1),\n        SchemaGen.scan_files(TEST_DIR / \"test_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        SchemaGen.scan_files(TEST_DIR / \"test_credit_bureau_a_2_*.parquet\", 2),\n        SchemaGen.scan_files(TEST_DIR / \"test_credit_bureau_b_2.parquet\", 2),\n    ],\n}\n\ndf_test: pl.DataFrame = (\n    SchemaGen.join_dataframes(**data_store)\n    .pipe(transform_cols)\n    .pipe(handle_dates)\n    .select([col for col in df_train.columns if col != \"target\"])\n    .pipe(Utility.reduce_memory_usage, \"df_test\")\n)\n\ndel data_store\ngc.collect()\n\nprint(f\"Test data shape: {df_test.shape}\")\n\ndf_test.write_parquet(\"test_final.parquet\", compression=\"lz4\")","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:03:28.617457Z","iopub.execute_input":"2024-05-14T11:03:28.617799Z","iopub.status.idle":"2024-05-14T11:03:33.173514Z","shell.execute_reply.started":"2024-05-14T11:03:28.617773Z","shell.execute_reply":"2024-05-14T11:03:33.172557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train, cat_cols = Utility.to_pandas(df_train)\ndf_test, cat_cols = Utility.to_pandas(df_test, cat_cols)","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:03:33.174819Z","iopub.execute_input":"2024-05-14T11:03:33.175197Z","iopub.status.idle":"2024-05-14T11:04:09.298487Z","shell.execute_reply.started":"2024-05-14T11:03:33.175164Z","shell.execute_reply":"2024-05-14T11:04:09.297668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class VotingModel(BaseEstimator, ClassifierMixin):\n    \"\"\"\n    A voting ensemble model that combines predictions from multiple estimators.\n\n    Parameters:\n    - estimators (list): List of base estimators.\n\n    Attributes:\n    - estimators (list): List of base estimators.\n\n    Methods:\n    - fit(X, y=None): Fit the model to the training data.\n    - predict(X): Predict class labels for samples.\n    - predict_proba(X): Predict class probabilities for samples.\n    \"\"\"\n\n    def __init__(self, estimators: list[BaseEstimator]):\n        \"\"\"\n        Initialize the VotingModel with a list of base estimators.\n\n        Args:\n        - estimators (list): List of base estimators.\n        \"\"\"\n        super().__init__()\n        self.estimators = estimators\n\n    def fit(self, X, y=None):\n        \"\"\"\n        Fit the model to the training data.\n\n        Args:\n        - X: Input features.\n        - y: Target labels (ignored).\n\n        Returns:\n        - self: Returns the instance itself.\n        \"\"\"\n        return self\n\n    def predict(self, X):\n        \"\"\"\n        Predict class labels for samples.\n\n        Args:\n        - X: Input features.\n\n        Returns:\n        - numpy.ndarray: Predicted class labels.\n        \"\"\"\n        y_preds = [estimator.predict(X) for estimator in self.estimators]\n        return np.mean(y_preds, axis=0)\n\n    def predict_proba(self, X):\n        \"\"\"\n        Predict class probabilities for samples.\n\n        Args:\n        - X: Input features.\n\n        Returns:\n        - numpy.ndarray: Predicted class probabilities.\n        \"\"\"\n        y_preds = [estimator.predict_proba(X) for estimator in self.estimators]\n        return np.mean(y_preds, axis=0)","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:04:09.301944Z","iopub.execute_input":"2024-05-14T11:04:09.302362Z","iopub.status.idle":"2024-05-14T11:04:09.311529Z","shell.execute_reply.started":"2024-05-14T11:04:09.302322Z","shell.execute_reply":"2024-05-14T11:04:09.3105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_subm: pd.DataFrame = pd.read_csv(ROOT / \"sample_submission.csv\")\ndf_subm = df_subm.set_index(\"case_id\")\n\ndevice: str = \"gpu\"\nest_cnt: int = 6000\n\nDRY_RUN = True if df_subm.shape[0] == 10 else False\nif DRY_RUN:\n    device = \"cpu\"\n    df_train = df_train.iloc[:50000]\n    est_cnt: int = 600\n\nprint(device)","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:04:09.312463Z","iopub.execute_input":"2024-05-14T11:04:09.312754Z","iopub.status.idle":"2024-05-14T11:04:09.334102Z","shell.execute_reply.started":"2024-05-14T11:04:09.312731Z","shell.execute_reply":"2024-05-14T11:04:09.333109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = df_train.drop(columns=[\"target\", \"case_id\", \"week_num\"])\ny = df_train[\"target\"]\n\nweeks = df_train[\"week_num\"]\n\ndel df_train\ngc.collect()\n\ncv = StratifiedGroupKFold(n_splits=5, shuffle=False)\n\nparam_list = [\n    {\n    \"boosting_type\": \"gbdt\",\n    \"colsample_bynode\": 0.8,\n    \"colsample_bytree\": 0.8,\n    \"device\": device,\n    \"extra_trees\": True,\n    \"learning_rate\": 0.013176075130689708,\n    \"max_depth\": 18,\n    \"metric\": \"auc\",\n    \"n_estimators\": 2000,\n    \"num_leaves\": 136,\n    \"objective\": \"binary\",\n    \"random_state\": 42,\n    'l1_regularization': 0.2402824722241209,\n    'l2_regularization': 5.039720588724357,\n    \"verbose\": -1,\n    },\n    # 定义其他四组参数\n    {    \n    \"boosting_type\": \"gbdt\",\n    \"colsample_bynode\": 0.8,\n    \"colsample_bytree\": 0.8,\n    \"device\": device,\n    \"extra_trees\": True,\n    \"learning_rate\": 0.010758890653308963,\n    \"l1_regularization\": 0.5379959045557414,\n    \"l2_regularization\": 5.785980480198053,\n    \"max_depth\": 14,\n    \"metric\": \"auc\",\n    \"n_estimators\": 2000,\n    \"num_leaves\": 136,\n    \"objective\": \"binary\",\n    \"random_state\": 42,\n    \"verbose\": -1,\n    },\n    {'boosting_type': 'gbdt', 'objective': 'binary', 'metric': 'auc', 'max_depth': 14, 'learning_rate': 0.02524838507372372, 'n_estimators': 2000, 'colsample_bytree': 0.8, 'colsample_bynode': 0.8, 'verbose': -1, 'random_state': 42, 'reg_alpha': 0.7171162007340861, 'reg_lambda': 3.9222194256656953, 'extra_trees': True, 'gpu_use_dp': True, 'num_leaves': 105, 'device': device,},\n    {'boosting_type': 'gbdt', 'objective': 'binary', 'metric': 'auc', 'max_depth': 12, 'learning_rate': 0.014151837756638192, 'n_estimators': 2000, 'colsample_bytree': 0.8, 'colsample_bynode': 0.8, 'verbose': -1, 'random_state': 42, 'reg_alpha': 0.17603404379281745, 'reg_lambda': 4.42025484269527, 'extra_trees': True, 'gpu_use_dp': True, 'num_leaves': 114, 'device': device,},\n    {'boosting_type': 'gbdt', 'objective': 'binary', 'metric': 'auc', 'max_depth': 19, 'learning_rate': 0.01767656540717102, 'n_estimators': 2000, 'colsample_bytree': 0.8, 'colsample_bynode': 0.8, 'verbose': -1, 'random_state': 42, 'reg_alpha': 0.09197393334346743, 'reg_lambda': 0.6975119148372935, 'extra_trees': True, 'gpu_use_dp': True, 'num_leaves': 146, 'device': device,}\n]\n\n\n\n\nfitted_models_cat = []\nfitted_models_lgb = []\n\ncv_scores_cat = []\ncv_scores_lgb = []\n\niter_cnt = 0\nfor idx_train, idx_valid in cv.split(X, y, groups=weeks):\n    X_train, y_train = X.iloc[idx_train], y.iloc[idx_train]\n    X_valid, y_valid = X.iloc[idx_valid], y.iloc[idx_valid]\n\n    train_pool = Pool(X_train, y_train, cat_features=cat_cols)\n    val_pool = Pool(X_valid, y_valid, cat_features=cat_cols)\n\n    clf = CatBoostClassifier(\n        best_model_min_trees = 1200,\n        boosting_type = \"Plain\",\n        eval_metric = \"AUC\",\n        iterations = est_cnt,\n        learning_rate = 0.05,\n        l2_leaf_reg = 10,\n        max_leaves = 64,\n        random_seed = 42,\n        task_type = \"GPU\",\n        use_best_model = True\n    )\n\n    clf.fit(train_pool, eval_set=val_pool, verbose=False)\n    fitted_models_cat.append(clf)\n\n    y_pred_valid = clf.predict_proba(X_valid)[:, 1]\n    auc_score = roc_auc_score(y_valid, y_pred_valid)\n    cv_scores_cat.append(auc_score)\n\n    X_train[cat_cols] = X_train[cat_cols].astype(\"category\")\n    X_valid[cat_cols] = X_valid[cat_cols].astype(\"category\")\n    \n    params = param_list[iter_cnt % 5]\n    \n    model = lgb.LGBMClassifier(**params)\n\n    model.fit(\n        X_train,\n        y_train,\n        eval_set=[(X_valid, y_valid)],\n        callbacks=[lgb.log_evaluation(100), lgb.early_stopping(100)],\n    )\n    fitted_models_lgb.append(model)\n\n    y_pred_valid = model.predict_proba(X_valid)[:, 1]\n    auc_score = roc_auc_score(y_valid, y_pred_valid)\n    cv_scores_lgb.append(auc_score)\n\n    iter_cnt += 1\n\nmodel = VotingModel(fitted_models_cat + fitted_models_lgb)\n\nprint(f\"\\nCV AUC scores for CatBoost: {cv_scores_cat}\")\nprint(f\"Maximum CV AUC score for Catboost: {max(cv_scores_cat)}\", end=\"\\n\\n\")\n\n\nprint(f\"CV AUC scores for LGBM: {cv_scores_lgb}\")\nprint(f\"Maximum CV AUC score for LGBM: {max(cv_scores_lgb)}\", end=\"\\n\\n\")\n\ndel X, y\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:04:09.335909Z","iopub.execute_input":"2024-05-14T11:04:09.336249Z","iopub.status.idle":"2024-05-14T11:13:52.361944Z","shell.execute_reply.started":"2024-05-14T11:04:09.336216Z","shell.execute_reply":"2024-05-14T11:13:52.36104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test: pd.DataFrame = df_test.drop(columns=[\"week_num\"]).set_index(\"case_id\")\n\nX_test[cat_cols] = X_test[cat_cols].astype(\"category\")\n\ny_pred: pd.Series = pd.Series(model.predict_proba(X_test)[:, 1], index=X_test.index)\n\ndf_subm[\"score\"] = y_pred\n\ndisplay(df_subm)\n\ndf_subm.to_csv(\"submission.csv\")\n\ndel X_test, y_pred, df_subm\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-05-14T11:13:52.363325Z","iopub.execute_input":"2024-05-14T11:13:52.363699Z","iopub.status.idle":"2024-05-14T11:13:53.103382Z","shell.execute_reply.started":"2024-05-14T11:13:52.363671Z","shell.execute_reply":"2024-05-14T11:13:53.102263Z"},"trusted":true},"execution_count":null,"outputs":[]}]}