{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30715,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Home credit default risk (IITD, Prof. Niladri)","metadata":{}},{"cell_type":"code","source":"import os\nfrom pathlib import Path\n\n# IO + data\nimport pandas as pd\nimport polars as pl\nimport numpy as np\n\n# viz\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-07-30T22:20:14.546612Z","iopub.execute_input":"2024-07-30T22:20:14.547028Z","iopub.status.idle":"2024-07-30T22:20:15.981934Z","shell.execute_reply.started":"2024-07-30T22:20:14.546990Z","shell.execute_reply":"2024-07-30T22:20:15.980763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"class CFG:\n    NULL_CUTOFF = 0.8  # 0.8\n    COL_MAX_UNIQUE_VALUES = 200  # 200\n    SAMPLE_FRAC = 0.4  # works at 0.5, not at 0.76\n    SEED = 13","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:15.984367Z","iopub.execute_input":"2024-07-30T22:20:15.985027Z","iopub.status.idle":"2024-07-30T22:20:15.991337Z","shell.execute_reply.started":"2024-07-30T22:20:15.984982Z","shell.execute_reply":"2024-07-30T22:20:15.989924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll be splitting our work in three phases:\n1. Data reading + analysis\n2. Data cleaning + preprocessing\n3. Model(s) and output\n\n# ETL\n## Reading the data","metadata":{}},{"cell_type":"code","source":"def get_available_files():\n    files = []\n    for dirname, _, filenames in os.walk('/kaggle/input'):\n        for filename in filenames:\n            files.append(os.path.join(dirname, filename))\n    return files\n\nget_available_files()","metadata":{"_kg_hide-output":true,"scrolled":true,"execution":{"iopub.status.busy":"2024-07-30T22:20:15.993032Z","iopub.execute_input":"2024-07-30T22:20:15.993494Z","iopub.status.idle":"2024-07-30T22:20:16.063348Z","shell.execute_reply.started":"2024-07-30T22:20:15.993453Z","shell.execute_reply":"2024-07-30T22:20:16.062237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have our input data available in two formats - `csv` and `parquet`. `parquet` is a format used for tabular data storage and provides decent compression and enhanced performance. Let's compare the sizes of our files in either format.\n\nhttps://en.wikipedia.org/wiki/Apache_Parquet","metadata":{}},{"cell_type":"code","source":"def get_folder_size(path='.'):\n    size = 0\n    for entry in os.scandir(path):\n        if entry.is_file():\n            size += entry.stat().st_size\n        elif entry.is_dir():\n            size += get_folder_size(entry.path)\n    return size\n\n{\n    'csv': f'{get_folder_size(\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files\") * 1e-9:.2f}GB',\n    'parquet': f'{get_folder_size(\"/kaggle/input/home-credit-credit-risk-model-stability/parquet_files\") * 1e-9:.2f}GB',\n}","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.064758Z","iopub.execute_input":"2024-07-30T22:20:16.065183Z","iopub.status.idle":"2024-07-30T22:20:16.079181Z","shell.execute_reply.started":"2024-07-30T22:20:16.065148Z","shell.execute_reply":"2024-07-30T22:20:16.077946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hence proved. We're using `parquet`. :)\n\nWe also have a question of the library we'll be using to actually read our files. We have the standard `pandas` library, but there's a cool new competitor called `polars` which offers much better performance. We'll use that. It's also written in `rust`, which is awesome. :)\n\nhttps://blog.jetbrains.com/dataspell/2023/08/polars-vs-pandas-what-s-the-difference/","metadata":{}},{"cell_type":"code","source":"ROOT = Path('/kaggle/input/home-credit-credit-risk-model-stability')\n\nTRAIN_DIR = ROOT / 'parquet_files' / 'train'\nTEST_DIR = ROOT / 'parquet_files' / 'test'","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.083058Z","iopub.execute_input":"2024-07-30T22:20:16.083946Z","iopub.status.idle":"2024-07-30T22:20:16.089239Z","shell.execute_reply.started":"2024-07-30T22:20:16.083900Z","shell.execute_reply":"2024-07-30T22:20:16.087952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As for feature definitions, the team has given us a `feature_definitions.csv` file with our column names, and a list of transforms have been provided as suffixes. Let's see if there are any columns that don't need to be transformed accordingly.","metadata":{}},{"cell_type":"code","source":"feature_definitions_df = pl.read_csv('/kaggle/input/home-credit-credit-risk-model-stability/feature_definitions.csv')\n\nfeature_definitions_df.filter(~feature_definitions_df['Variable'].str.ends_with('P')\n                              & ~feature_definitions_df['Variable'].str.ends_with('M')\n                              & ~feature_definitions_df['Variable'].str.ends_with('A')\n                              & ~feature_definitions_df['Variable'].str.ends_with('D')\n                              & ~feature_definitions_df['Variable'].str.ends_with('T')\n                              & ~feature_definitions_df['Variable'].str.ends_with('L'))","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.091107Z","iopub.execute_input":"2024-07-30T22:20:16.091678Z","iopub.status.idle":"2024-07-30T22:20:16.340800Z","shell.execute_reply.started":"2024-07-30T22:20:16.091580Z","shell.execute_reply":"2024-07-30T22:20:16.339651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above, we can see that every column (with the exception of `score_940`, estimate of a client's trustworthiness) has to be transformed. The transformations are according to their suffixes:\n- `P`: Transform DPD (Days past due)\n- `M`: Masking categories\n- `A`: Transform amount\n- `D`: Transform date\n- `T`: Unspecified Transform\n- `L`: Unspecified Transform\n\nOther than that, for certain files with `depth`, we have to apply aggregate transforms. These columns are titled `num_group1` and `num_group2`. It'd be best to come up with a set of aggregates to use on all dataframes as we load them.\n\n## Aggregates and transforms (`Aggregates`)","metadata":{}},{"cell_type":"code","source":"class Aggregates:\n    # TODO: alias the cols to \"{col}_{'max'|'min'|'median'|'whatever'}\" so that it's sorted correctly when we see it in the output\n    aggr_fns = {\n        'median': pl.median,\n        'min': pl.min,\n        'max': pl.max,\n        'count': pl.count,\n    }\n    \n    @staticmethod\n    def __get_aggrs_for_fn(aggr_fn, cols):\n        if aggr_fn not in Aggregates.aggr_fns:\n            raise Exception(f'Unknown aggregator \"{aggr_fn}\"')\n        \n        return [Aggregates.aggr_fns[aggr_fn](col).alias(f'{aggr_fn}_{col}') for col in cols]\n    \n    @staticmethod\n    def __get_aggrs(aggr_fns, cols):\n        aggrs = []\n        for aggr_fn in aggr_fns:\n            aggrs += Aggregates.__get_aggrs_for_fn(aggr_fn, cols)\n\n        return aggrs\n    \n    @staticmethod\n    def num_aggr(df, aggr_fns=['median']):\n        cols = [col for col in df.columns if col[-1] in (\"P\", \"A\")]\n        return Aggregates.__get_aggrs(aggr_fns, cols)\n    \n    @staticmethod\n    def date_aggr(df, aggr_fns=['median']):\n        cols = [col for col in df.columns if col[-1] in (\"D\")]\n        return Aggregates.__get_aggrs(aggr_fns, cols)\n    \n    @staticmethod\n    def str_aggr(df, aggr_fns=['median']):\n        cols = [col for col in df.columns if col[-1] in (\"M\")]\n        return Aggregates.__get_aggrs(aggr_fns, cols)\n    \n    @staticmethod\n    def other_aggr(df, aggr_fns=['median']):\n        cols = [col for col in df.columns if col[-1] in (\"T\", \"L\")]\n        return Aggregates.__get_aggrs(aggr_fns, cols)\n    \n    @staticmethod\n    def count_aggr(df, aggr_fns=['median']):\n        cols = [col for col in df.columns if \"num_group\" in col]\n        return Aggregates.__get_aggrs(aggr_fns, cols)\n    \n    @staticmethod\n    def get_aggregates(df, aggr_fns=['median']):\n        aggrs = Aggregates.num_aggr(df, aggr_fns) + \\\n                Aggregates.date_aggr(df, aggr_fns) + \\\n                Aggregates.str_aggr(df, aggr_fns) + \\\n                Aggregates.other_aggr(df, aggr_fns) + \\\n                Aggregates.count_aggr(df, aggr_fns)\n\n        return aggrs","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.342470Z","iopub.execute_input":"2024-07-30T22:20:16.342949Z","iopub.status.idle":"2024-07-30T22:20:16.357271Z","shell.execute_reply.started":"2024-07-30T22:20:16.342910Z","shell.execute_reply":"2024-07-30T22:20:16.356129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below are a set of methods for when we're loading our data\n\n## Dataframe operations (`DataFrameOps`)\nThe above include setting datatypes, processing dates, dropping string cols with too many unique values and reducing memory usage (since we are easily running out of memory)","metadata":{}},{"cell_type":"code","source":"from pprint import pprint\n\nclass DataFrameOps:\n    dropped_train_cols = None\n    \n    @staticmethod\n    def set_dtypes(df):\n        for col in df.columns:\n            if col in ['case_id', 'WEEK_NUM', 'num_group1', 'num_group2']:\n                df = df.with_columns(pl.col(col).cast(pl.Int64))\n            elif col in ['date_decision']:\n                df = df.with_columns(pl.col(col).cast(pl.Date))\n            elif col[-1] in ('P', 'A'):\n                df = df.with_columns(pl.col(col).cast(pl.Float64))\n            elif col[-1] in ('M',):\n                df = df.with_columns(pl.col(col).cast(pl.String))\n            elif col[-1] in ('D',):\n                df = df.with_columns(pl.col(col).cast(pl.Date))\n        return df\n    \n    @staticmethod\n    def process_dates(df):\n        processed_cols = []\n        for col in df.columns:\n            if col[-1] in ('D',):\n                processed_cols.append(col)\n                df = df.with_columns(pl.col(col) - pl.col('date_decision'))\n                df = df.with_columns(pl.col(col).dt.total_days())\n        \n        print(f'DataFrameOps.process_dates / processed:')\n        pprint(processed_cols)\n\n        df = df.drop('date_decision')\n        return df\n\n    @staticmethod\n    def drop_string_cols_with_numerous_values(df, is_test):\n        if is_test:\n            print(f'DataFrameOps.drop_string_cols_with_numerous_values / dropping for train values of:')\n            pprint(DataFrameOps.dropped_train_cols)\n\n            df = df.drop(list(DataFrameOps.dropped_train_cols))\n            return df\n        \n        DataFrameOps.dropped_train_cols = {}\n        \n        for col in df.columns:\n            if col in ['target', 'case_id', 'WEEK_NUM']:\n                continue\n            \n            if df[col].dtype != pl.String:\n                continue\n\n            freq = df[col].n_unique()\n            if freq == 1 or freq > CFG.COL_MAX_UNIQUE_VALUES:\n                DataFrameOps.dropped_train_cols[col] = freq\n        df.drop(list(DataFrameOps.dropped_train_cols))\n            \n        print(f'DataFrameOps.drop_string_cols_with_numerous_values:')\n        pprint(DataFrameOps.dropped_train_cols)\n        \n        return df\n    \n    @staticmethod\n    \n    def reduce_memory_usage_pl(df):\n        \"\"\"\n            Reduce memory usage by polars dataframe {df} by changing its data types.\n            pandas version: https://www.kaggle.com/code/arjanso/reducing-dataframe-memory-size-by-65 \n            polars version: https://www.kaggle.com/code/demche/polars-memory-usage-optimization\n        \"\"\"\n        start_memory_mb = round(df.estimated_size('mb'), 2)\n\n        numeric_int_types = [pl.Int8, pl.Int16, pl.Int32, pl.Int64]\n        numeric_flo_types = [pl.Float32, pl.Float64]\n\n        for col in df.columns:\n            col_type = df[col].dtype\n            if col_type not in numeric_int_types or col_type not in numeric_flo_types:\n                continue\n            \n            c_min, c_max = df[col].min(), df[col].max()\n            if c_min is None or c_max is None:\n                continue\n\n            if col_type in numeric_int_types:\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df = df.with_columns(df[col].cast(pl.Int8))\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df = df.with_columns(df[col].cast(pl.Int16))\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df = df.with_columns(df[col].cast(pl.Int32))\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df = df.with_columns(df[col].cast(pl.Int64))\n            elif col_type in numeric_flo_types:\n                if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df = df.with_columns(df[col].cast(pl.Float32))\n\n        end_memory_mb = round(df.estimated_size('mb'), 2)\n        print(f'DataFrameOps.reduce_memory_usage_pl / reduced memory usage from {start_memory_mb}MB to {end_memory_mb}MB')\n\n        return df","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.358748Z","iopub.execute_input":"2024-07-30T22:20:16.359222Z","iopub.status.idle":"2024-07-30T22:20:16.381175Z","shell.execute_reply.started":"2024-07-30T22:20:16.359193Z","shell.execute_reply":"2024-07-30T22:20:16.379910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, once the aggregations and table ops are set, we can load our files\n\n## Reading the files (`ParquetReader`)","metadata":{}},{"cell_type":"code","source":"class ParquetReader:\n    @staticmethod\n    def read(path, depth=None):\n        df = pl.read_parquet(path)\n        df = df.pipe(DataFrameOps.set_dtypes)\n        \n        if depth in [1, 2]: # could replace with `if depth`\n            df = df.group_by('case_id').agg(Aggregates.get_aggregates(df))\n\n        return df\n    \n    @staticmethod\n    def read_multiple(paths, depth=None):\n        chunks = [ParquetReader.read(path, depth) for path in paths]\n        \n        df = pl.concat(chunks, how='vertical_relaxed')\n        # drop any duplicates\n        df = df.unique(subset=['case_id'])\n        \n        return df","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.382589Z","iopub.execute_input":"2024-07-30T22:20:16.382956Z","iopub.status.idle":"2024-07-30T22:20:16.397924Z","shell.execute_reply.started":"2024-07-30T22:20:16.382926Z","shell.execute_reply":"2024-07-30T22:20:16.396571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And we should somehow merge the loaded parquet files.\n## Feature engineer and merge dataframes","metadata":{}},{"cell_type":"code","source":"def feature_engineer_and_merge_dataframes(base, depth_0, depth_1, depth_2, is_test):\n    base = base.with_columns(\n        year_decision = pl.col('date_decision').dt.year(),\n        month_decision = pl.col('date_decision').dt.month(),\n        weekday_decision = pl.col('date_decision').dt.weekday())\n\n    for i, df in enumerate(depth_0 + depth_1 + depth_2):\n        base = base.join(df, how='left', on='case_id', suffix=f'_{i}')\n        \n    base = base.pipe(DataFrameOps.process_dates)\n#     base = base.pipe(DataFrameOps.drop_string_cols_with_numerous_values, is_test)\n    base = base.pipe(DataFrameOps.reduce_memory_usage_pl)\n    \n    return base","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.399362Z","iopub.execute_input":"2024-07-30T22:20:16.399765Z","iopub.status.idle":"2024-07-30T22:20:16.410050Z","shell.execute_reply.started":"2024-07-30T22:20:16.399728Z","shell.execute_reply":"2024-07-30T22:20:16.408920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"LETS REEAAAAADDDDDDDDDD THEM DATAS","metadata":{}},{"cell_type":"code","source":"train_df = feature_engineer_and_merge_dataframes(\n    base=ParquetReader.read(TRAIN_DIR / 'train_base.parquet'),\n    depth_0=[\n        ParquetReader.read_multiple([\n            TRAIN_DIR / 'train_static_0_0.parquet',\n            TRAIN_DIR / 'train_static_0_1.parquet',\n        ]),\n        ParquetReader.read(TRAIN_DIR / 'train_static_cb_0.parquet'),\n    ],\n    depth_1=[\n        ParquetReader.read_multiple([\n            TRAIN_DIR / 'train_applprev_1_0.parquet',\n            TRAIN_DIR / 'train_applprev_1_1.parquet',\n        ], depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_other_1.parquet', depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_tax_registry_a_1.parquet', depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_tax_registry_b_1.parquet', depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_tax_registry_c_1.parquet', depth=1),\n        ParquetReader.read_multiple([\n            TRAIN_DIR / 'train_credit_bureau_a_1_0.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_1_1.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_1_2.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_1_3.parquet',\n        ], depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_credit_bureau_b_1.parquet', depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_deposit_1.parquet', depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_person_1.parquet', depth=1),\n        ParquetReader.read(TRAIN_DIR / 'train_debitcard_1.parquet', depth=1),\n    ],\n    depth_2=[\n        ParquetReader.read(TRAIN_DIR / 'train_applprev_2.parquet', depth=2),\n        ParquetReader.read(TRAIN_DIR / 'train_person_2.parquet', depth=2),\n        ParquetReader.read_multiple([\n            TRAIN_DIR / 'train_credit_bureau_a_2_0.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_1.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_2.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_3.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_4.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_5.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_6.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_7.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_8.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_9.parquet',\n            TRAIN_DIR / 'train_credit_bureau_a_2_10.parquet',\n        ], depth=2),\n        ParquetReader.read(TRAIN_DIR / 'train_credit_bureau_b_2.parquet', depth=2),\n    ],\n    is_test=False\n)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:20:16.411259Z","iopub.execute_input":"2024-07-30T22:20:16.411750Z","iopub.status.idle":"2024-07-30T22:23:37.094220Z","shell.execute_reply.started":"2024-07-30T22:20:16.411719Z","shell.execute_reply":"2024-07-30T22:23:37.093145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Sample\ndef do_sample(df):\n    original_length = len(df)\n    df = df.sample(fraction=CFG.SAMPLE_FRAC, seed=CFG.SEED, shuffle=True)\n    new_length = len(df)\n    \n    print(f'We\\'ve gone from {original_length} rows to {new_length} rows')\n    \n    return df\n\ntrain_df = do_sample(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:23:37.095616Z","iopub.execute_input":"2024-07-30T22:23:37.095946Z","iopub.status.idle":"2024-07-30T22:23:37.871716Z","shell.execute_reply.started":"2024-07-30T22:23:37.095919Z","shell.execute_reply":"2024-07-30T22:23:37.870381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I can't be 100% sure that the above works flawlessly, but we do what we can with what we've got.\n\n# EDA\n_(we're finally here, phew)_\n\nLet's first start by plotting how well formed our data is - how many nullish columns do we have?\n## Seeing nullish columns","metadata":{}},{"cell_type":"code","source":"def plot_nulls(df):\n    fig, ax = plt.subplots(figsize=(72, 12))\n\n    def get_null_stats(col):\n        null_count = df[col].is_null().sum()\n        return { 'name': col, 'null_count': null_count, 'null_percentage': null_count / len(df) }\n\n    null_df = pd.DataFrame([get_null_stats(col) for col in df.columns])\n    null_df = null_df.sort_values('null_percentage', ascending=False)\n    \n    sns.barplot(null_df, x='name', y='null_percentage')\n    plt.axhline(y=CFG.NULL_CUTOFF, linestyle='-')\n    plt.xticks(rotation=90)\n    \n    ax.set_xlabel('Column name')\n    ax.set_ylabel('Percentage of nulls')\n    ax.set_title('Percentage of nulls per column')\n\n    return null_df\n\ntrain_null_stats_df = plot_nulls(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:23:37.873103Z","iopub.execute_input":"2024-07-30T22:23:37.873438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_null_stats_df[train_null_stats_df['null_percentage'] > CFG.NULL_CUTOFF]","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:26:29.314842Z","iopub.execute_input":"2024-07-30T22:26:29.315304Z","iopub.status.idle":"2024-07-30T22:26:29.330483Z","shell.execute_reply.started":"2024-07-30T22:26:29.315270Z","shell.execute_reply":"2024-07-30T22:26:29.329428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see above, a lot of columns have simply null values, and since they don't provide much information, we should be good to drop these entirely.\n\nSome such columns could be informational, but we're just dumping them for now. In the case that we need to get the absolute best performance out of our system, we'd look at columns individually and figure out whether we can drop them or not.\n\nFor now, we'll just drop all columns with more null values than specified in `CFG.NULL_CUTOFF`.","metadata":{}},{"cell_type":"code","source":"# TODO: this can be done in the loading phase itself, while we're doing much of the string removing math as well.\ndef drop_nullish_columns(df, null_stats_df):\n    original_df_shape = df.shape\n\n    for col in [colname for colname in null_stats_df[null_stats_df['null_percentage'] > CFG.NULL_CUTOFF]['name']]:\n        df = df.drop(col)\n\n    updated_df_shape = df.shape\n    \n    return df, original_df_shape, updated_df_shape\n\ntrain_df, old_shape, new_shape = drop_nullish_columns(train_df, train_null_stats_df)\n\nold_shape, new_shape","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:26:34.477532Z","iopub.execute_input":"2024-07-30T22:26:34.478001Z","iopub.status.idle":"2024-07-30T22:26:34.517978Z","shell.execute_reply.started":"2024-07-30T22:26:34.477962Z","shell.execute_reply":"2024-07-30T22:26:34.516543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Correlation matrix\nWe won't be able to find correlations between date types without transforming them to a different numerical column, so we'll drop them while calculating our correlation matrix.\n\nWe also won't be able to find correlations between string types without first converting those to categorical.","metadata":{}},{"cell_type":"code","source":"def plot_correlation_matrix(df):\n    # to_pandas because polars returns nulls if a single value in the column is null\n    corr = df.select(~pl.selectors.by_dtype(pl.Date, pl.String)).to_pandas().corr()\n    \n    _, (ax1) = plt.subplots(figsize=(60, 60))\n\n    sns.heatmap(corr, vmax=1, center=0, square=True,\n                mask=np.triu(np.ones_like(corr, dtype=bool)),\n                cmap=sns.diverging_palette(230, 20, as_cmap=True), ax=ax1)\n    \n    plt.show()\n\n    return corr\n\n_correlation_df = plot_correlation_matrix(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:26:37.941519Z","iopub.execute_input":"2024-07-30T22:26:37.941973Z","iopub.status.idle":"2024-07-30T22:27:03.087293Z","shell.execute_reply.started":"2024-07-30T22:26:37.941940Z","shell.execute_reply":"2024-07-30T22:27:03.085754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see above, there is a significant negative correlation of `target` with `pctinstlsallpaidearl3d_427L`, `numinstpaidearly3dest_4493216L`, `numinstmatpaidtearly2d_4499204L`, `numinstpaidearly5dobd_4499205L`, `numinstpaidearlyest_4493214L`, `numinstpaidearly3d_3546850L`, `numinstlallpaidearly3d_817L`, and a positive correlation with `avgmaxdpdlast9m_3716943P`, `numinstlswithdpd10_728L`, `pctinstlsallpaidlat10d_839L`, `pctinstlsallpaidlate6d_3546844L`, `pctinstlsallpaidlate4d_3546849L` and `pctinstlsallpaidlate1d_3546856L`.\n\nThis is expected behavior. If the user has paid most of their installments early, then there's a higher chance that they are good with their credit and won't default - hence the negative correlation.\nAu contraire, if all their installments are paid late, there's a higher chance they are bad with their credit and are more likely to default - hence the positive correlation.","metadata":{}},{"cell_type":"code","source":"print('Highest negative correlations:')\n_correlation_df['target'].sort_values().head()","metadata":{"execution":{"iopub.status.idle":"2024-07-30T22:27:03.100806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Highest positive correlations:')\n_correlation_df['target'].sort_values(ascending=False).head(6)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:03.102155Z","iopub.execute_input":"2024-07-30T22:27:03.102513Z","iopub.status.idle":"2024-07-30T22:27:03.120104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot values over time","metadata":{}},{"cell_type":"code","source":"def plot_values_over_time(df):\n    pd_df = train_df.to_pandas()\n    ax = sns.countplot(pd_df, x='MONTH', hue='target')\n\n    plt.ylim(0, 60000)\n    plt.xticks(rotation=90)\n    for container in ax.containers:\n        ax.bar_label(container, rotation=90, fontsize=6)\n        \n    ax.legend(['Repaid', 'Defaulted'])\n    ax.set_xlabel('Date (YYYYMM)')\n    ax.set_ylabel('Count')\n    ax.set_title('Date vs number of cases')\n\nplot_values_over_time(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:09.629250Z","iopub.execute_input":"2024-07-30T22:27:09.629652Z","iopub.status.idle":"2024-07-30T22:27:10.512391Z","shell.execute_reply.started":"2024-07-30T22:27:09.629620Z","shell.execute_reply":"2024-07-30T22:27:10.511209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One really cool thing to see is the data that we have is right around covid - the loan defaults and repayments drop drastically ever since 201912, and dip to almost zero during the first wave. Fairly certain this will have an influence our training, so we'll probably need to stratify our data in phase 2. Let's have a look at how many defaults and repayments we have in 201912 and in 202004, so as to compare.\n\n## Stats around covid-19 compared to a normal month","metadata":{}},{"cell_type":"code","source":"{\n    '201912': {\n        'defaulted': train_df.filter((train_df['MONTH'] == '201912') & (train_df['target'] == 1)).shape[0],\n        'repaid': train_df.filter((train_df['MONTH'] == '201912') & (train_df['target'] == 0)).shape[0]\n    },\n    '202004': {\n        'defaulted': train_df.filter((train_df['MONTH'] == '202004') & (train_df['target'] == 1)).shape[0],\n        'repaid': train_df.filter((train_df['MONTH'] == '202004') & (train_df['target'] == 0)).shape[0]\n    }\n}","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:10.514563Z","iopub.execute_input":"2024-07-30T22:27:10.515041Z","iopub.status.idle":"2024-07-30T22:27:10.548408Z","shell.execute_reply.started":"2024-07-30T22:27:10.515000Z","shell.execute_reply":"2024-07-30T22:27:10.547294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And what about the default rate over time?\n## Rate of defaulting over time","metadata":{}},{"cell_type":"code","source":"def plot_default_percentage_with_time(df):\n    _df = df.group_by('MONTH').agg([pl.len().alias('total'), pl.sum('target').alias('defaults')])\n    _df = _df.with_columns(pl.col('MONTH').cast(pl.Utf8))\n    _df = _df.with_columns(default_rate=_df['defaults'] * 100 / _df['total'])\n    _df = _df.to_pandas().sort_values('MONTH')\n\n    fig, ax = plt.subplots()\n    \n    sns.lineplot(_df, x='MONTH', y='default_rate')\n    plt.xticks(rotation=90)\n    \n    ax.legend(['Default rate'])\n    ax.set_xlabel('Date (YYYYMM)')\n    ax.set_ylabel('Default rate %')\n    ax.set_title('Date vs default rate')\n    \nplot_default_percentage_with_time(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:10.601332Z","iopub.execute_input":"2024-07-30T22:27:10.602185Z","iopub.status.idle":"2024-07-30T22:27:11.030368Z","shell.execute_reply.started":"2024-07-30T22:27:10.602143Z","shell.execute_reply":"2024-07-30T22:27:11.029092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above, we can see that the default rate slowly increases from August of 2019 onwards, all the way to April of 2020, which is when it stops. This is probably due to the banks being very selective to whom they give loans.\n## Rate of defaulting as a function of the applicant's age","metadata":{}},{"cell_type":"code","source":"def plot_age_to_default(df):\n    _df = df.select(['birthdate_574D', 'target']).drop_nulls()\n    _df = _df.with_columns(pl.col('birthdate_574D') / -365)\n    _df = _df.with_columns(pl.col('birthdate_574D').cut(range(0, 100, 5), labels=[f'{i}' for i in range(0, 101, 5)]).alias('age'))\n    _df = _df.group_by('age').agg([pl.len().alias('count'), pl.sum('target').alias('defaults')])\n    _df = _df.with_columns(default_rate=_df['defaults'] * 100 / _df['count'])\n    _df = _df.to_pandas().sort_values('age')\n        \n    fig, ax = plt.subplots()\n        \n    sns.lineplot(_df, x='age', y='default_rate')\n    \n    ax.legend(['Default rate'])\n    ax.set_xlabel('Age')\n    ax.set_ylabel('Default rate %')\n    ax.set_title('Age vs default rate')\n\nplot_age_to_default(train_df)\n# list()","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:11.181551Z","iopub.execute_input":"2024-07-30T22:27:11.181979Z","iopub.status.idle":"2024-07-30T22:27:11.746017Z","shell.execute_reply.started":"2024-07-30T22:27:11.181945Z","shell.execute_reply":"2024-07-30T22:27:11.744787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above, we can see that the likelihood of defaulting decreases with an increase in age. This is likely due to the fact that someone older usually has a longer career behind them, and is less likely to default due to it.\n## Rate of defaulting as a function of the applicant's current debt","metadata":{}},{"cell_type":"code","source":"def plot_current_debt_to_default(df, nb_bins=None, bin_size=50000):\n    _df = df.select(['currdebt_22A', 'target']).drop_nulls()\n    \n    min_debt, max_debt = int(df['currdebt_22A'].min()), int(df['currdebt_22A'].max())\n    bin_size = (max_debt - min_debt) // nb_bins if nb_bins is not None else bin_size\n\n    cut = list(range(min_debt, max_debt, bin_size))\n    cut_labels = [f'{i}' for i in range(min_debt, max_debt + bin_size, bin_size)]\n\n    _df = _df.with_columns(pl.col('currdebt_22A').cut(cut, labels=cut_labels).alias('debt'))\n    _df = _df.group_by('debt').agg([pl.len().alias('count'), pl.sum('target').alias('defaults')])\n    _df = _df.with_columns(default_rate=_df['defaults'] * 100 / _df['count'])\n    _df = _df.to_pandas().sort_values('debt')\n    \n    fig, ax = plt.subplots()\n    \n    sns.lineplot(_df, x='debt', y='default_rate')\n    plt.xticks(rotation=90)\n    \n    ax.legend(['Default rate'])\n    ax.set_xlabel('Current debt')\n    ax.set_ylabel('Default rate %')\n    ax.set_title('Current debt vs default rate')\n\nplot_current_debt_to_default(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:12.649426Z","iopub.execute_input":"2024-07-30T22:27:12.649815Z","iopub.status.idle":"2024-07-30T22:27:13.095016Z","shell.execute_reply.started":"2024-07-30T22:27:12.649786Z","shell.execute_reply":"2024-07-30T22:27:13.093811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above, we can see that the likelihood of defaulting increases as the amount of debt increases. Fairly standard observation. The rate of defaulting is very low in the case of very high debts. This is probably due to the banks not approving loan requests for people with higher debts.\n## Rate of defaulting as a function of the applicant's income","metadata":{}},{"cell_type":"code","source":"def plot_income_to_default(df, nb_bins=None, bin_size=10000):\n    _df = df.select(['maininc_215A', 'target']).drop_nulls()\n    \n    fig, ax = plt.subplots()\n    \n    min_debt, max_debt = int(df['maininc_215A'].min()), int(df['maininc_215A'].max())\n    bin_size = (max_debt - min_debt) // nb_bins if nb_bins is not None else bin_size\n\n    cut = list(range(min_debt, max_debt, bin_size))\n    cut_labels = [f'{i}' for i in range(min_debt, max_debt + bin_size, bin_size)]\n\n    _df = _df.with_columns(pl.col('maininc_215A').cut(cut, labels=cut_labels).alias('income'))\n    _df = _df.group_by('income').agg([pl.len().alias('count'), pl.sum('target').alias('approved')])\n    _df = _df.with_columns(approval_rate=_df['approved'] * 100 / _df['count'])\n    _df = _df.to_pandas().sort_values('income')\n        \n    sns.lineplot(_df, x='income', y='approval_rate')\n    plt.xticks(rotation=90)\n\n    ax.legend(['Default rate'])\n    ax.set_xlabel('Annual income')\n    ax.set_ylabel('Default rate %')\n    ax.set_title('Annual income vs default rate')\n\nplot_income_to_default(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:13.421269Z","iopub.execute_input":"2024-07-30T22:27:13.421769Z","iopub.status.idle":"2024-07-30T22:27:13.852204Z","shell.execute_reply.started":"2024-07-30T22:27:13.421732Z","shell.execute_reply":"2024-07-30T22:27:13.850824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see above, the rate of defaulting is higher among low income groups, and then stabilizes around an income of \\\\$40000. Strangely, it increases around an income level of \\\\$120000. This could be due to higher income individuals opting for more expensive properties and larger loan amounts.\n## Combined plot showing income and debt against likelihood of defaulting","metadata":{}},{"cell_type":"code","source":"%%time\n\ndef plot_income_and_debt_against_default_rate(df, nb_samples=20000):\n    _df = df.select(['maininc_215A', 'currdebt_22A', 'target']).sample(nb_samples, shuffle=True, seed=CFG.SEED).drop_nulls()\n    \n    f, ax = plt.subplots(figsize=(12, 8))\n    ax.set_aspect('equal')\n    \n    plt.xlim(_df['currdebt_22A'].min(), _df['currdebt_22A'].max())\n    plt.ylim(_df['maininc_215A'].min(), _df['maininc_215A'].max())\n    \n    default_df = _df.filter(pl.col('target') == 1)\n    n_defau_df = _df.filter(pl.col('target') == 0)\n\n    sns.kdeplot(n_defau_df, x='currdebt_22A', y='maininc_215A', color='green', levels=10)\n    sns.kdeplot(default_df, x='currdebt_22A', y='maininc_215A', color='red', levels=2)\n\n    sns.histplot(n_defau_df, x='currdebt_22A', y='maininc_215A', color='green', bins=25)\n    sns.histplot(default_df, x='currdebt_22A', y='maininc_215A', color='red', bins=25)\n    \n    sns.scatterplot(n_defau_df, x='currdebt_22A', y='maininc_215A', color='green', s=5, label='Repaid')\n    sns.scatterplot(default_df, x='currdebt_22A', y='maininc_215A', color='red', s=15, label='Defaulted')\n    \n    ax.legend()\n    ax.set_xlabel('Current debt')\n    ax.set_ylabel('Annual income')\n    ax.set_title('Current debt vs annual income')\n    \nplot_income_and_debt_against_default_rate(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:13.973135Z","iopub.execute_input":"2024-07-30T22:27:13.974233Z","iopub.status.idle":"2024-07-30T22:27:25.649239Z","shell.execute_reply.started":"2024-07-30T22:27:13.974189Z","shell.execute_reply":"2024-07-30T22:27:25.647947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above, we can see that there is a higher likelihood of a person defaulting if they're in a lower income group with a potentially higher debt. High income individuals have a lower chance of defaulting unless their debt is also equally high.\n\nThe higher the income, the lesser the chance of defaulting. The lower income groups with higher debts are at a very high chance of defaulting as compared to the rest of the sample.\n\nLet's go ahead and plot the frequency of an applicant paying their installments early and paying their installments late against their chance of defaulting. These two columns are highly negatively and positively correlated with the target, and should provide some really good information.\n## Combined plot showing the number of installments paid early and paid late against likelihood of defaulting","metadata":{}},{"cell_type":"code","source":"%%time\n\ndef plot_early_and_late_payments_against_default_rate(df, nb_samples=20000):\n    _df = df.select(['pctinstlsallpaidearl3d_427L', 'pctinstlsallpaidlate1d_3546856L', 'target']).sample(nb_samples, shuffle=True, seed=CFG.SEED).drop_nulls()\n    \n    _df = _df.filter(pl.col('pctinstlsallpaidearl3d_427L') <= 1)\n    _df = _df.filter(pl.col('pctinstlsallpaidlate1d_3546856L') <= 1)\n    \n    f, ax = plt.subplots(figsize=(8, 8))\n    \n    plt.ylim(0, 1)\n    plt.xlim(0, 1)\n\n    default_df = _df.filter(pl.col('target') == 1)\n    n_defau_df = _df.filter(pl.col('target') == 0)\n\n    sns.kdeplot(n_defau_df, y='pctinstlsallpaidearl3d_427L', x='pctinstlsallpaidlate1d_3546856L', color='green', levels=10)\n    sns.kdeplot(default_df, y='pctinstlsallpaidearl3d_427L', x='pctinstlsallpaidlate1d_3546856L', color='red', levels=2)\n\n    sns.histplot(n_defau_df, y='pctinstlsallpaidearl3d_427L', x='pctinstlsallpaidlate1d_3546856L', color='green', bins=25)\n    sns.histplot(default_df, y='pctinstlsallpaidearl3d_427L', x='pctinstlsallpaidlate1d_3546856L', color='red', bins=25)\n\n    sns.scatterplot(n_defau_df, y='pctinstlsallpaidearl3d_427L', x='pctinstlsallpaidlate1d_3546856L', color='green', s=3, label='Repaid')\n    sns.scatterplot(default_df, y='pctinstlsallpaidearl3d_427L', x='pctinstlsallpaidlate1d_3546856L', color='red', s=15, label='Defaulted')\n    \n    ax.legend()\n    ax.set_xlabel('Percentage of installments paid late')\n    ax.set_ylabel('Percentage of installments paid early')\n    ax.set_title('Installments paid late vs paid early')\n    pass\n\nplot_early_and_late_payments_against_default_rate(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:25.651487Z","iopub.execute_input":"2024-07-30T22:27:25.651912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model, training and output\n## Getting clean `X` and `y` from our dataframes","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import OneHotEncoder, StandardScaler\n\ndef transform_get_X_y(df, col_transformer=None):\n    reserved_cols = ['target', 'MONTH', 'WEEK_NUM', 'case_id']\n    \n    y = df['target'].to_numpy() if 'target' in df.columns else None\n    df = df.drop(reserved_cols)\n    \n    cat_cols = [col for col in df.columns if df[col].dtype == pl.String]\n    num_cols = [col for col in df.columns if df[col].dtype != pl.String]\n\n    if not col_transformer:\n        col_transformer = ColumnTransformer(transformers=[\n            ('imputer', SimpleImputer(), num_cols),  # this doesn't work with categorical data, so only nums. onehotencoder will deal with categorica nulls.\n            ('cat_cols', OneHotEncoder(), cat_cols),\n            ('num_cols', StandardScaler(), num_cols),\n        ], n_jobs=-1).fit(df.to_pandas())\n    \n    X = col_transformer.transform(df.to_pandas())\n    \n    return X, y, col_transformer\n\n_X, _y, _transformer = transform_get_X_y(train_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:34.331211Z","iopub.execute_input":"2024-07-30T22:27:34.331558Z","iopub.status.idle":"2024-07-30T22:27:47.861375Z","shell.execute_reply.started":"2024-07-30T22:27:34.331526Z","shell.execute_reply":"2024-07-30T22:27:47.859898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Getting the train and validation sets for our train data","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ndef get_train_val_split(X, y):\n    return train_test_split(_X, y, test_size=0.2, random_state=CFG.SEED)\n\n_X_train, _X_val, _y_train, _y_val = get_train_val_split(_X, _y)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:53.501265Z","iopub.execute_input":"2024-07-30T22:27:53.502344Z","iopub.status.idle":"2024-07-30T22:27:53.819793Z","shell.execute_reply.started":"2024-07-30T22:27:53.502293Z","shell.execute_reply":"2024-07-30T22:27:53.818587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model definition","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom xgboost import XGBClassifier\n\nimport warnings\nwarnings.filterwarnings('default')\n\neval_metrics = ['error', 'logloss', 'auc', 'pre']\n\nmodel = XGBClassifier(random_state=CFG.SEED, eval_metric=eval_metrics, early_stopping_rounds=15)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:27:56.497218Z","iopub.execute_input":"2024-07-30T22:27:56.498145Z","iopub.status.idle":"2024-07-30T22:27:56.505485Z","shell.execute_reply.started":"2024-07-30T22:27:56.498107Z","shell.execute_reply":"2024-07-30T22:27:56.504230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model training","metadata":{}},{"cell_type":"code","source":"%%time\n\nmodel.fit(_X_train, _y_train,\n          eval_set=[(_X_train, _y_train), (_X_val, _y_val)],\n          verbose=True)\n\n_y_preds = model.predict(_X_val)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:28:06.086754Z","iopub.execute_input":"2024-07-30T22:28:06.087167Z","iopub.status.idle":"2024-07-30T22:28:19.892032Z","shell.execute_reply.started":"2024-07-30T22:28:06.087139Z","shell.execute_reply":"2024-07-30T22:28:19.891060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plotting model performance","metadata":{}},{"cell_type":"code","source":"from pprint import pprint\n\ndef plot_model_metrics(metrics):\n    fig, axs = plt.subplots(len(eval_metrics) // 2, 2, figsize=(10 * (len(eval_metrics) // 2), 5 * 2))\n\n    epochs = len(metrics['validation_0'][eval_metrics[0]])\n    x_axis = range(epochs)\n    \n    for i, metric in enumerate(eval_metrics):\n        axs[i // 2][i % 2].plot(x_axis, metrics['validation_0'][metric], label='Train')\n        axs[i // 2][i % 2].plot(x_axis, metrics['validation_1'][metric], label='Test')\n        axs[i // 2][i % 2].set_ylabel(metric)\n        axs[i // 2][i % 2].set_xlabel('epochs')\n        axs[i // 2][i % 2].legend()\n\n    plt.show()\n\nplot_model_metrics(model.evals_result())","metadata":{"execution":{"iopub.status.busy":"2024-07-30T22:36:43.726465Z","iopub.execute_input":"2024-07-30T22:36:43.726904Z","iopub.status.idle":"2024-07-30T22:36:44.624176Z","shell.execute_reply.started":"2024-07-30T22:36:43.726856Z","shell.execute_reply":"2024-07-30T22:36:44.622968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's do a small sanity check to ensure that our model isn't going for all loans paid or something like that - that it's actually creating both defaulted and non defaulted outputs for our predictions.","metadata":{}},{"cell_type":"code","source":"pl.Series(_y_preds).value_counts()","metadata":{"execution":{"iopub.status.busy":"2024-07-30T18:11:13.063886Z","iopub.execute_input":"2024-07-30T18:11:13.064385Z","iopub.status.idle":"2024-07-30T18:11:13.080083Z","shell.execute_reply.started":"2024-07-30T18:11:13.064339Z","shell.execute_reply":"2024-07-30T18:11:13.078337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test dataset and submission\nGetting rid of train data to reduce memory usage","metadata":{}},{"cell_type":"code","source":"del train_df\ndel _X_train, _y_train\ndel _X_val, _y_val","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Load and transform the test data to get `X`.","metadata":{}},{"cell_type":"code","source":"test_df = feature_engineer_and_merge_dataframes(\n    is_test=True,\n    base=ParquetReader.read(TEST_DIR / 'test_base.parquet'),\n    depth_0=[\n        ParquetReader.read_multiple([\n            TEST_DIR / 'test_static_0_0.parquet',\n            TEST_DIR / 'test_static_0_1.parquet',\n            TEST_DIR / 'test_static_0_2.parquet',\n        ]),\n        ParquetReader.read(TEST_DIR / 'test_static_cb_0.parquet'),\n    ],\n    depth_1=[\n        ParquetReader.read_multiple([\n            TEST_DIR / 'test_applprev_1_0.parquet',\n            TEST_DIR / 'test_applprev_1_1.parquet',\n            TEST_DIR / 'test_applprev_1_2.parquet',\n        ], depth=1),\n        ParquetReader.read(TEST_DIR / 'test_other_1.parquet', depth=1),\n        ParquetReader.read(TEST_DIR / 'test_tax_registry_a_1.parquet', depth=1),\n        ParquetReader.read(TEST_DIR / 'test_tax_registry_b_1.parquet', depth=1),\n        ParquetReader.read(TEST_DIR / 'test_tax_registry_c_1.parquet', depth=1),\n        ParquetReader.read_multiple([\n            TEST_DIR / 'test_credit_bureau_a_1_0.parquet',\n            TEST_DIR / 'test_credit_bureau_a_1_1.parquet',\n            TEST_DIR / 'test_credit_bureau_a_1_2.parquet',\n            TEST_DIR / 'test_credit_bureau_a_1_3.parquet',\n            TEST_DIR / 'test_credit_bureau_a_1_4.parquet',\n        ], depth=1),\n        ParquetReader.read(TEST_DIR / 'test_credit_bureau_b_1.parquet', depth=1),\n        ParquetReader.read(TEST_DIR / 'test_deposit_1.parquet', depth=1),\n        ParquetReader.read(TEST_DIR / 'test_person_1.parquet', depth=1),\n        ParquetReader.read(TEST_DIR / 'test_debitcard_1.parquet', depth=1),\n    ],\n    depth_2=[\n        ParquetReader.read(TEST_DIR / 'test_applprev_2.parquet', depth=2),\n        ParquetReader.read(TEST_DIR / 'test_person_2.parquet', depth=2),\n        ParquetReader.read_multiple([\n            TEST_DIR / 'test_credit_bureau_a_2_0.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_1.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_2.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_3.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_4.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_5.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_6.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_7.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_8.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_9.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_10.parquet',\n            TEST_DIR / 'test_credit_bureau_a_2_11.parquet',\n        ], depth=2),\n        ParquetReader.read(TEST_DIR / 'test_credit_bureau_b_2.parquet', depth=2),\n    ]\n)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T18:16:51.953597Z","iopub.execute_input":"2024-07-30T18:16:51.954475Z","iopub.status.idle":"2024-07-30T18:16:52.227897Z","shell.execute_reply.started":"2024-07-30T18:16:51.954431Z","shell.execute_reply":"2024-07-30T18:16:52.226528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dropping columns we dropped in our train set","metadata":{}},{"cell_type":"code","source":"test_df, _, _ = drop_nullish_columns(test_df, train_null_stats_df)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T18:17:05.731310Z","iopub.execute_input":"2024-07-30T18:17:05.732404Z","iopub.status.idle":"2024-07-30T18:17:05.774965Z","shell.execute_reply.started":"2024-07-30T18:17:05.732362Z","shell.execute_reply":"2024-07-30T18:17:05.773708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Getting `X_test`","metadata":{}},{"cell_type":"code","source":"_X_test, _, _transformer = transform_get_X_y(test_df, col_transformer=_transformer)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T18:17:06.346623Z","iopub.execute_input":"2024-07-30T18:17:06.347053Z","iopub.status.idle":"2024-07-30T18:17:08.450681Z","shell.execute_reply.started":"2024-07-30T18:17:06.347017Z","shell.execute_reply":"2024-07-30T18:17:08.448476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building `y_pred` using the model we trained earlier","metadata":{}},{"cell_type":"code","source":"_test_preds = model.predict(_X_test)","metadata":{"execution":{"iopub.status.busy":"2024-07-30T18:17:10.308213Z","iopub.execute_input":"2024-07-30T18:17:10.308719Z","iopub.status.idle":"2024-07-30T18:17:10.322213Z","shell.execute_reply.started":"2024-07-30T18:17:10.308676Z","shell.execute_reply":"2024-07-30T18:17:10.320772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generating the submission file","metadata":{}},{"cell_type":"code","source":"submission = pl.DataFrame({'case_id': test_df['case_id'], 'score': _test_preds})\n\nsubmission.to_pandas().to_csv('submission.csv', index=False)\n\nsubmission","metadata":{"execution":{"iopub.status.busy":"2024-07-30T18:17:11.958410Z","iopub.execute_input":"2024-07-30T18:17:11.958846Z","iopub.status.idle":"2024-07-30T18:17:11.968859Z","shell.execute_reply.started":"2024-07-30T18:17:11.958811Z","shell.execute_reply":"2024-07-30T18:17:11.967545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fin. :)","metadata":{}}]}