{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7602123,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Setup","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport polars as pl\nimport polars.selectors as cs\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nimport missingno as mn","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:55:52.289329Z","iopub.execute_input":"2024-03-13T00:55:52.289962Z","iopub.status.idle":"2024-03-13T00:55:54.769829Z","shell.execute_reply.started":"2024-03-13T00:55:52.289903Z","shell.execute_reply":"2024-03-13T00:55:54.768019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before proceeding, we prepare some utility functions.","metadata":{}},{"cell_type":"markdown","source":"Feature description is found in `feature_definitions.csv`.","metadata":{}},{"cell_type":"code","source":"df_feature_definitions = pl.read_csv('/kaggle/input/home-credit-credit-risk-model-stability/feature_definitions.csv')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:55:58.427948Z","iopub.execute_input":"2024-03-13T00:55:58.428531Z","iopub.status.idle":"2024-03-13T00:55:58.666719Z","shell.execute_reply.started":"2024-03-13T00:55:58.428494Z","shell.execute_reply":"2024-03-13T00:55:58.665559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(50)\n    display(df_feature_definitions.head())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:55:58.955340Z","iopub.execute_input":"2024-03-13T00:55:58.955756Z","iopub.status.idle":"2024-03-13T00:55:58.980353Z","shell.execute_reply.started":"2024-03-13T00:55:58.955725Z","shell.execute_reply":"2024-03-13T00:55:58.978883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Select specific features.","metadata":{}},{"cell_type":"code","source":"def get_feature_definitions(columns):\n    return pl.DataFrame({'Variable': columns}).join(\n        df_feature_definitions,\n        on = 'Variable',\n        how = 'left',\n    )","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:02.259979Z","iopub.execute_input":"2024-03-13T00:56:02.260407Z","iopub.status.idle":"2024-03-13T00:56:02.267937Z","shell.execute_reply.started":"2024-03-13T00:56:02.260374Z","shell.execute_reply":"2024-03-13T00:56:02.266437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(50)\n    display(get_feature_definitions(['actualdpd_943P', 'addres_zip_823M']))","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:03.932351Z","iopub.execute_input":"2024-03-13T00:56:03.932776Z","iopub.status.idle":"2024-03-13T00:56:04.008110Z","shell.execute_reply.started":"2024-03-13T00:56:03.932745Z","shell.execute_reply":"2024-03-13T00:56:04.006915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some features should be of dtype `date` (or `datetime`).\n\n> Various predictors were transformed, therefore we have the following notation for similar groups of transformations\n> \n> - P - Transform DPD (Days past due)\n> - M - Masking categories\n> - A - Transform amount\n> - D - Transform date\n> - T - Unspecified Transform\n> - L - Unspecified Transform\n> \n> Please note that transformations within a group are denoted by a capital letter at the end of the predictor name (e.g., maxdbddpdtollast6m_4187119P). We hope that this will simplify the manipulation with predictors.\n\nWe cast columns ending `D` to `datetime` at read time.\nAlso note that we use `pl.scan_parquet()` to save memory.","metadata":{}},{"cell_type":"code","source":"def scan_data_and_parse_dates(parquet_file):\n    return pl.scan_parquet(parquet_file).with_columns(\n        pl.col(r'^.*D$').str.to_datetime(),\n    )","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:06.173475Z","iopub.execute_input":"2024-03-13T00:56:06.173877Z","iopub.status.idle":"2024-03-13T00:56:06.180017Z","shell.execute_reply.started":"2024-03-13T00:56:06.173846Z","shell.execute_reply":"2024-03-13T00:56:06.178619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Function to plot (hist+kde & box) features of a specific transformation group.","metadata":{}},{"cell_type":"code","source":"def plot_numerical(df, group, max_samples = 1000000):\n    X = df.select(pl.col(f'^.*{group}$')).collect()\n    if len(X) > max_samples:\n        X = X.sample(max_samples, seed = 0)\n    n = X.shape[1]\n    fig, axes = plt.subplots(n, 2, figsize = (16, n * 4), squeeze = False)\n    for i, c in enumerate(X.iter_columns()):\n        x = c.to_pandas()\n        sns.histplot(x = x, bins = 50, kde = True, ax = axes[i, 0])\n        sns.boxplot(y = x, ax = axes[i, 1])\n    fig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:06.822627Z","iopub.execute_input":"2024-03-13T00:56:06.823038Z","iopub.status.idle":"2024-03-13T00:56:06.833117Z","shell.execute_reply.started":"2024-03-13T00:56:06.823006Z","shell.execute_reply":"2024-03-13T00:56:06.831255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Function to plot (count & pie) features of a specific transformation group.","metadata":{}},{"cell_type":"code","source":"def plot_categorical(df, group, max_categories = 30, max_samples = 1000000):\n    X = df.select(pl.col(f'^.*{group}$')).collect()\n    if len(X) > max_samples:\n        X = X.sample(max_samples, seed = 0)\n    n = X.shape[1]\n    fig, axes = plt.subplots(n, 2, figsize = (16, n * 4), squeeze = False)\n    for i, c in enumerate(X.iter_columns()):\n        x = c.to_pandas()\n        s = x.value_counts()\n        if len(s) > max_categories:\n            other_count = s[max_categories:].sum()\n            s = s[:max_categories]\n            s['Other'] = other_count\n        sns.barplot(x = s.index, y = s, ax = axes[i, 0])\n        axes[i, 1].pie(s, labels = s.index)\n    fig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:07.263243Z","iopub.execute_input":"2024-03-13T00:56:07.264428Z","iopub.status.idle":"2024-03-13T00:56:07.274360Z","shell.execute_reply.started":"2024-03-13T00:56:07.264384Z","shell.execute_reply":"2024-03-13T00:56:07.273144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Function to plot correlation among a group.","metadata":{}},{"cell_type":"code","source":"def plot_corr(df, group, max_samples = 1000000):\n    X = df.select(pl.col(f'^.*{group}$')).collect()\n    if len(X) > max_samples:\n        X = X.sample(max_samples, seed = 0)\n    corr = X.to_pandas(use_pyarrow_extension_array = True).corr()\n    w = len(corr)\n    h = int(w * 0.8)\n    _, ax = plt.subplots(figsize = (w, h))\n    sns.heatmap(corr, ax = ax, vmin = -1, vmax = 1, annot = True, cmap = 'coolwarm')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:07.692804Z","iopub.execute_input":"2024-03-13T00:56:07.693221Z","iopub.status.idle":"2024-03-13T00:56:07.702045Z","shell.execute_reply.started":"2024-03-13T00:56:07.693189Z","shell.execute_reply":"2024-03-13T00:56:07.700436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Function to plot null values.","metadata":{}},{"cell_type":"code","source":"def plot_null(df, group, max_samples = 1000000):\n    X = df.select(pl.col(f'^.*{group}$')).collect()\n    if len(X) > max_samples:\n        X = X.sample(max_samples, seed = 0)\n    n = max(X.shape[1], 8)\n    X = X.to_pandas(use_pyarrow_extension_array = True)\n    fig, axes = plt.subplots(2, figsize = (n, int(1.6 * n)))\n    mn.matrix(X, ax = axes[0], sparkline = False)\n    mn.heatmap(X, ax = axes[1])\n    fig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:08.158262Z","iopub.execute_input":"2024-03-13T00:56:08.159500Z","iopub.status.idle":"2024-03-13T00:56:08.169853Z","shell.execute_reply.started":"2024-03-13T00:56:08.159447Z","shell.execute_reply":"2024-03-13T00:56:08.168107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Function to plot numerical variables, grouped by target.","metadata":{}},{"cell_type":"code","source":"def plot_numerical_by_target(df_base, df, group, max_samples = 1000000):\n    df_base = df_base.lazy()\n    X = df_base.join(\n        df, on = 'case_id', how = 'left',\n    ).select('target', pl.col(f'^.*{group}$')).collect()\n    if len(X) > max_samples:\n        X = X.sample(max_samples, seed = 0)\n    y = X['target'].to_pandas()\n    X = X.drop('target')\n    n = X.shape[1]\n    fig, axes = plt.subplots(n, 2, figsize = (16, n * 4), squeeze = False)\n    for i, c in enumerate(X.iter_columns()):\n        x = c.to_pandas()\n        sns.violinplot(x = y, y = x, orient = 'v', order = [0, 1], ax = axes[i, 0])\n        sns.stripplot(x = y, y = x, orient = 'v', order = [0, 1], ax = axes[i, 1],\n                      marker = '.')\n    fig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:08.655241Z","iopub.execute_input":"2024-03-13T00:56:08.656462Z","iopub.status.idle":"2024-03-13T00:56:08.667931Z","shell.execute_reply.started":"2024-03-13T00:56:08.656413Z","shell.execute_reply":"2024-03-13T00:56:08.666684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Base table","metadata":{}},{"cell_type":"code","source":"df_train_base = pl.read_parquet('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_base.parquet')\ndf_train_base = df_train_base.with_columns(\n    pl.col('date_decision').str.to_datetime(),\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:09.807685Z","iopub.execute_input":"2024-03-13T00:56:09.808185Z","iopub.status.idle":"2024-03-13T00:56:11.178849Z","shell.execute_reply.started":"2024-03-13T00:56:09.808150Z","shell.execute_reply":"2024-03-13T00:56:11.176989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_base.head())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:11.181516Z","iopub.execute_input":"2024-03-13T00:56:11.181931Z","iopub.status.idle":"2024-03-13T00:56:11.191347Z","shell.execute_reply.started":"2024-03-13T00:56:11.181896Z","shell.execute_reply":"2024-03-13T00:56:11.189835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_base.describe())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:11.267684Z","iopub.execute_input":"2024-03-13T00:56:11.268090Z","iopub.status.idle":"2024-03-13T00:56:11.415403Z","shell.execute_reply.started":"2024-03-13T00:56:11.268050Z","shell.execute_reply":"2024-03-13T00:56:11.413767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_base['target'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:11.512044Z","iopub.execute_input":"2024-03-13T00:56:11.513254Z","iopub.status.idle":"2024-03-13T00:56:11.567621Z","shell.execute_reply.started":"2024-03-13T00:56:11.513200Z","shell.execute_reply":"2024-03-13T00:56:11.566120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_base['target'].value_counts().with_columns(pl.col('count') / len(df_train_base)))","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:11.741930Z","iopub.execute_input":"2024-03-13T00:56:11.742356Z","iopub.status.idle":"2024-03-13T00:56:11.793293Z","shell.execute_reply.started":"2024-03-13T00:56:11.742323Z","shell.execute_reply":"2024-03-13T00:56:11.791260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The target is imbalanced. Only 3% of the entries have defaulted.","metadata":{}},{"cell_type":"markdown","source":"`MONTH` column is redundant. Indeed, `year * 100 + month == MONTH`.","metadata":{}},{"cell_type":"code","source":"display(df_train_base.select(\n    (\n        pl.col('date_decision').dt.year() * 100 + pl.col('date_decision').dt.month()\n         == pl.col('MONTH')\n    ).all()))","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:12.501620Z","iopub.execute_input":"2024-03-13T00:56:12.502091Z","iopub.status.idle":"2024-03-13T00:56:12.560082Z","shell.execute_reply.started":"2024-03-13T00:56:12.502055Z","shell.execute_reply":"2024-03-13T00:56:12.558882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`WEEK_NUM` is also redundant. It is the week counting from `2019-01-01`.","metadata":{}},{"cell_type":"code","source":"display(df_train_base.group_by('WEEK_NUM').agg(pl.col('date_decision').min()).sort('WEEK_NUM').head())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:13.084531Z","iopub.execute_input":"2024-03-13T00:56:13.085147Z","iopub.status.idle":"2024-03-13T00:56:13.120662Z","shell.execute_reply.started":"2024-03-13T00:56:13.085107Z","shell.execute_reply":"2024-03-13T00:56:13.119375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_base.select(\n    (\n        (pl.col('date_decision') - pl.date(2019, 1, 1)).dt.total_days() // 7\n        == pl.col('WEEK_NUM')\n    ).all()\n))","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:13.350225Z","iopub.execute_input":"2024-03-13T00:56:13.350878Z","iopub.status.idle":"2024-03-13T00:56:13.426590Z","shell.execute_reply.started":"2024-03-13T00:56:13.350844Z","shell.execute_reply":"2024-03-13T00:56:13.425259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`case_id` is a unique identifier.","metadata":{}},{"cell_type":"code","source":"print(df_train_base['case_id'].n_unique() == len(df_train_base))","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:13.862115Z","iopub.execute_input":"2024-03-13T00:56:13.863286Z","iopub.status.idle":"2024-03-13T00:56:13.888861Z","shell.execute_reply.started":"2024-03-13T00:56:13.863226Z","shell.execute_reply":"2024-03-13T00:56:13.887335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot date.","metadata":{}},{"cell_type":"code","source":"sns.histplot(x = df_train_base['date_decision'].to_pandas(),\n             bins = 50,\n             kde = True);","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:14.270110Z","iopub.execute_input":"2024-03-13T00:56:14.270888Z","iopub.status.idle":"2024-03-13T00:56:21.610645Z","shell.execute_reply.started":"2024-03-13T00:56:14.270850Z","shell.execute_reply":"2024-03-13T00:56:21.609486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The number of cases significantly decreased around `2020-04`. Might affect CV.","metadata":{}},{"cell_type":"code","source":"sns.violinplot(y = df_train_base.select((pl.col('date_decision') - pl.date(2019, 1, 1)).dt.total_days())['date_decision'].to_pandas(),\n               x = df_train_base['target'].to_pandas(),\n               orient = 'v');","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:21.612727Z","iopub.execute_input":"2024-03-13T00:56:21.613124Z","iopub.status.idle":"2024-03-13T00:56:25.133746Z","shell.execute_reply.started":"2024-03-13T00:56:21.613093Z","shell.execute_reply":"2024-03-13T00:56:25.132305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.lineplot(df_train_base.group_by('WEEK_NUM').agg(pl.col('target').mean()).to_pandas(),\n             x = 'WEEK_NUM', y = 'target');","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:25.135237Z","iopub.execute_input":"2024-03-13T00:56:25.136399Z","iopub.status.idle":"2024-03-13T00:56:25.432581Z","shell.execute_reply.started":"2024-03-13T00:56:25.136347Z","shell.execute_reply":"2024-03-13T00:56:25.431554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# static_0","metadata":{}},{"cell_type":"markdown","source":"There are two files for static depth 0.","metadata":{}},{"cell_type":"code","source":"df_train_static_0_0 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_static_0_0.parquet')\ndf_train_static_0_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_static_0_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:25.435356Z","iopub.execute_input":"2024-03-13T00:56:25.436543Z","iopub.status.idle":"2024-03-13T00:56:25.474183Z","shell.execute_reply.started":"2024-03-13T00:56:25.436499Z","shell.execute_reply":"2024-03-13T00:56:25.472454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"They have the same schema.","metadata":{}},{"cell_type":"code","source":"print(df_train_static_0_0.schema == df_train_static_0_1.schema)","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:25.476494Z","iopub.execute_input":"2024-03-13T00:56:25.476976Z","iopub.status.idle":"2024-03-13T00:56:25.484792Z","shell.execute_reply.started":"2024-03-13T00:56:25.476935Z","shell.execute_reply":"2024-03-13T00:56:25.483370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we may scan all tables of the same kind at the same time and concatenate them.\n`pl.scan_parquet()` does this job given a globbing pattern.","metadata":{}},{"cell_type":"code","source":"df_train_static_0 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_static_0_*.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:25.486216Z","iopub.execute_input":"2024-03-13T00:56:25.487366Z","iopub.status.idle":"2024-03-13T00:56:25.508106Z","shell.execute_reply.started":"2024-03-13T00:56:25.487315Z","shell.execute_reply":"2024-03-13T00:56:25.507067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(200)\n    cfg.set_tbl_rows(-1)\n    display(get_feature_definitions(df_train_static_0.columns))","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:25.752654Z","iopub.execute_input":"2024-03-13T00:56:25.753576Z","iopub.status.idle":"2024-03-13T00:56:25.779511Z","shell.execute_reply.started":"2024-03-13T00:56:25.753538Z","shell.execute_reply":"2024-03-13T00:56:25.778515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_cols(-1)\n    display(df_train_static_0.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:27.381590Z","iopub.execute_input":"2024-03-13T00:56:27.382228Z","iopub.status.idle":"2024-03-13T00:56:27.803802Z","shell.execute_reply.started":"2024-03-13T00:56:27.382196Z","shell.execute_reply":"2024-03-13T00:56:27.802509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_cols(-1)\n    display(df_train_static_0.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:28.764869Z","iopub.execute_input":"2024-03-13T00:56:28.765253Z","iopub.status.idle":"2024-03-13T00:56:41.745683Z","shell.execute_reply.started":"2024-03-13T00:56:28.765222Z","shell.execute_reply":"2024-03-13T00:56:41.744733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_static_0.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:41.747420Z","iopub.execute_input":"2024-03-13T00:56:41.748150Z","iopub.status.idle":"2024-03-13T00:56:42.293064Z","shell.execute_reply.started":"2024-03-13T00:56:41.748113Z","shell.execute_reply":"2024-03-13T00:56:42.292013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_static_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:56:42.294441Z","iopub.execute_input":"2024-03-13T00:56:42.294826Z","iopub.status.idle":"2024-03-13T00:58:22.299126Z","shell.execute_reply.started":"2024-03-13T00:56:42.294795Z","shell.execute_reply":"2024-03-13T00:58:22.294376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_static_0, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.300944Z","iopub.status.idle":"2024-03-13T00:58:22.301576Z","shell.execute_reply.started":"2024-03-13T00:58:22.301264Z","shell.execute_reply":"2024-03-13T00:58:22.301323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_static_0, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.304255Z","iopub.status.idle":"2024-03-13T00:58:22.305055Z","shell.execute_reply.started":"2024-03-13T00:58:22.304798Z","shell.execute_reply":"2024-03-13T00:58:22.304822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_static_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.307167Z","iopub.status.idle":"2024-03-13T00:58:22.307717Z","shell.execute_reply.started":"2024-03-13T00:58:22.307443Z","shell.execute_reply":"2024-03-13T00:58:22.307462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_static_0, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.309622Z","iopub.status.idle":"2024-03-13T00:58:22.310149Z","shell.execute_reply.started":"2024-03-13T00:58:22.309910Z","shell.execute_reply":"2024-03-13T00:58:22.309929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_static_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.312431Z","iopub.status.idle":"2024-03-13T00:58:22.313245Z","shell.execute_reply.started":"2024-03-13T00:58:22.312838Z","shell.execute_reply":"2024-03-13T00:58:22.312870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_static_0, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.314735Z","iopub.status.idle":"2024-03-13T00:58:22.315200Z","shell.execute_reply.started":"2024-03-13T00:58:22.314981Z","shell.execute_reply":"2024-03-13T00:58:22.315000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_static_0, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.317687Z","iopub.status.idle":"2024-03-13T00:58:22.318211Z","shell.execute_reply.started":"2024-03-13T00:58:22.317968Z","shell.execute_reply":"2024-03-13T00:58:22.317989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_static_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.319983Z","iopub.status.idle":"2024-03-13T00:58:22.320518Z","shell.execute_reply.started":"2024-03-13T00:58:22.320251Z","shell.execute_reply":"2024-03-13T00:58:22.320294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_static_0, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T00:58:22.322397Z","iopub.status.idle":"2024-03-13T00:58:22.323674Z","shell.execute_reply.started":"2024-03-13T00:58:22.323131Z","shell.execute_reply":"2024-03-13T00:58:22.323163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Each group of tables can comprise one or more individual tables. If a group contains more than one table, they are divided based on `WEEK_NUM`.","metadata":{}},{"cell_type":"code","source":"import glob\n\nwith pl.Config() as cfg:\n    cfg.set_tbl_rows(-1)\n    for f in glob.iglob('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_static_0_*.parquet'):\n        display(\n            pl.scan_parquet(f)\n            .join(df_train_base.lazy(), on = 'case_id', how = 'left')\n            .select(pl.col('WEEK_NUM').unique())\n            .collect()\n        )","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:00:35.432376Z","iopub.execute_input":"2024-03-13T01:00:35.432787Z","iopub.status.idle":"2024-03-13T01:00:35.830403Z","shell.execute_reply.started":"2024-03-13T01:00:35.432757Z","shell.execute_reply":"2024-03-13T01:00:35.829336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# static_cb_0","metadata":{}},{"cell_type":"code","source":"df_train_static_cb_0 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_static_cb_0.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:01:32.762991Z","iopub.execute_input":"2024-03-13T01:01:32.763401Z","iopub.status.idle":"2024-03-13T01:01:32.783484Z","shell.execute_reply.started":"2024-03-13T01:01:32.763371Z","shell.execute_reply":"2024-03-13T01:01:32.782386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_rows(-1)\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_static_cb_0.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:26:32.924280Z","iopub.execute_input":"2024-02-13T01:26:32.925179Z","iopub.status.idle":"2024-02-13T01:26:32.941403Z","shell.execute_reply.started":"2024-02-13T01:26:32.925130Z","shell.execute_reply":"2024-02-13T01:26:32.940284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_static_cb_0.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:26:33.817781Z","iopub.execute_input":"2024-02-13T01:26:33.818183Z","iopub.status.idle":"2024-02-13T01:26:33.973924Z","shell.execute_reply.started":"2024-02-13T01:26:33.818151Z","shell.execute_reply":"2024-02-13T01:26:33.972940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_static_cb_0.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:26:56.812718Z","iopub.execute_input":"2024-02-13T01:26:56.813102Z","iopub.status.idle":"2024-02-13T01:26:59.856272Z","shell.execute_reply.started":"2024-02-13T01:26:56.813074Z","shell.execute_reply":"2024-02-13T01:26:59.855434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_static_cb_0.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:27:18.220195Z","iopub.execute_input":"2024-02-13T01:27:18.220923Z","iopub.status.idle":"2024-02-13T01:27:18.468297Z","shell.execute_reply.started":"2024-02-13T01:27:18.220885Z","shell.execute_reply":"2024-02-13T01:27:18.467287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_static_cb_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:27:43.757825Z","iopub.execute_input":"2024-02-13T01:27:43.758223Z","iopub.status.idle":"2024-02-13T01:27:53.249891Z","shell.execute_reply.started":"2024-02-13T01:27:43.758193Z","shell.execute_reply":"2024-02-13T01:27:53.248504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_static_cb_0, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:39:46.414348Z","iopub.execute_input":"2024-02-13T01:39:46.414890Z","iopub.status.idle":"2024-02-13T01:39:49.718400Z","shell.execute_reply.started":"2024-02-13T01:39:46.414838Z","shell.execute_reply":"2024-02-13T01:39:49.717470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_static_cb_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:33:15.240574Z","iopub.execute_input":"2024-02-13T04:33:15.240922Z","iopub.status.idle":"2024-02-13T04:33:15.751758Z","shell.execute_reply.started":"2024-02-13T04:33:15.240897Z","shell.execute_reply":"2024-02-13T04:33:15.750614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_static_cb_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:58:27.345756Z","iopub.execute_input":"2024-02-13T04:58:27.346824Z","iopub.status.idle":"2024-02-13T04:58:29.364783Z","shell.execute_reply.started":"2024-02-13T04:58:27.346780Z","shell.execute_reply":"2024-02-13T04:58:29.363659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_static_cb_0, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:59:19.919006Z","iopub.execute_input":"2024-02-13T04:59:19.919422Z","iopub.status.idle":"2024-02-13T04:59:22.401401Z","shell.execute_reply.started":"2024-02-13T04:59:19.919392Z","shell.execute_reply":"2024-02-13T04:59:22.400274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_static_cb_0, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T00:51:32.617872Z","iopub.execute_input":"2024-02-14T00:51:32.618337Z","iopub.status.idle":"2024-02-14T00:51:46.393208Z","shell.execute_reply.started":"2024-02-14T00:51:32.618302Z","shell.execute_reply":"2024-02-14T00:51:46.391955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# applprev_1","metadata":{}},{"cell_type":"code","source":"df_train_applprev_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_applprev_1_*.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:01:38.226239Z","iopub.execute_input":"2024-03-13T01:01:38.226757Z","iopub.status.idle":"2024-03-13T01:01:38.247050Z","shell.execute_reply.started":"2024-03-13T01:01:38.226720Z","shell.execute_reply":"2024-03-13T01:01:38.244981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    cfg.set_tbl_rows(-1)\n    display(get_feature_definitions(df_train_applprev_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:11:06.198879Z","iopub.execute_input":"2024-02-26T01:11:06.199741Z","iopub.status.idle":"2024-02-26T01:11:06.215875Z","shell.execute_reply.started":"2024-02-26T01:11:06.199700Z","shell.execute_reply":"2024-02-26T01:11:06.214234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_applprev_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:11:07.102907Z","iopub.execute_input":"2024-02-26T01:11:07.103348Z","iopub.status.idle":"2024-02-26T01:11:07.319419Z","shell.execute_reply.started":"2024-02-26T01:11:07.103316Z","shell.execute_reply":"2024-02-26T01:11:07.318301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_applprev_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:11:09.295091Z","iopub.execute_input":"2024-02-26T01:11:09.296404Z","iopub.status.idle":"2024-02-26T01:11:24.193744Z","shell.execute_reply.started":"2024-02-26T01:11:09.296349Z","shell.execute_reply":"2024-02-26T01:11:24.192692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_applprev_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:29:28.610532Z","iopub.execute_input":"2024-02-13T01:29:28.610935Z","iopub.status.idle":"2024-02-13T01:29:30.198245Z","shell.execute_reply.started":"2024-02-13T01:29:28.610904Z","shell.execute_reply":"2024-02-13T01:29:30.197119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(x = df_train_applprev_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:14:51.876194Z","iopub.execute_input":"2024-02-26T01:14:51.876672Z","iopub.status.idle":"2024-02-26T01:14:53.034950Z","shell.execute_reply.started":"2024-02-26T01:14:51.876637Z","shell.execute_reply":"2024-02-26T01:14:53.033390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- It is likely that the number of previous applications is clipped at `20`.","metadata":{}},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_applprev_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:30:13.806584Z","iopub.execute_input":"2024-02-13T01:30:13.806956Z","iopub.status.idle":"2024-02-13T01:33:54.666344Z","shell.execute_reply.started":"2024-02-13T01:30:13.806928Z","shell.execute_reply":"2024-02-13T01:33:54.664619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_applprev_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:33:54.698929Z","iopub.execute_input":"2024-02-13T01:33:54.699785Z","iopub.status.idle":"2024-02-13T01:34:43.692057Z","shell.execute_reply.started":"2024-02-13T01:33:54.699748Z","shell.execute_reply":"2024-02-13T01:34:43.690462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_applprev_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:39:59.368360Z","iopub.execute_input":"2024-02-13T01:39:59.368786Z","iopub.status.idle":"2024-02-13T01:40:13.324886Z","shell.execute_reply.started":"2024-02-13T01:39:59.368754Z","shell.execute_reply":"2024-02-13T01:40:13.323533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_applprev_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:34:03.137860Z","iopub.execute_input":"2024-02-13T04:34:03.138672Z","iopub.status.idle":"2024-02-13T04:34:05.608028Z","shell.execute_reply.started":"2024-02-13T04:34:03.138624Z","shell.execute_reply":"2024-02-13T04:34:05.606940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_applprev_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:34:09.137749Z","iopub.execute_input":"2024-02-13T04:34:09.138170Z","iopub.status.idle":"2024-02-13T04:34:09.611411Z","shell.execute_reply.started":"2024-02-13T04:34:09.138138Z","shell.execute_reply":"2024-02-13T04:34:09.610254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_applprev_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:00:10.529260Z","iopub.execute_input":"2024-02-13T05:00:10.530164Z","iopub.status.idle":"2024-02-13T05:00:16.359048Z","shell.execute_reply.started":"2024-02-13T05:00:10.530130Z","shell.execute_reply":"2024-02-13T05:00:16.357714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_applprev_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:00:18.225762Z","iopub.execute_input":"2024-02-13T05:00:18.226164Z","iopub.status.idle":"2024-02-13T05:00:19.465579Z","shell.execute_reply.started":"2024-02-13T05:00:18.226135Z","shell.execute_reply":"2024-02-13T05:00:19.464536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_applprev_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:00:25.067956Z","iopub.execute_input":"2024-02-13T05:00:25.069219Z","iopub.status.idle":"2024-02-13T05:00:28.919639Z","shell.execute_reply.started":"2024-02-13T05:00:25.069168Z","shell.execute_reply":"2024-02-13T05:00:28.918890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_applprev_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T00:53:43.863349Z","iopub.execute_input":"2024-02-14T00:53:43.863750Z","iopub.status.idle":"2024-02-14T00:54:40.444834Z","shell.execute_reply.started":"2024-02-14T00:53:43.863715Z","shell.execute_reply":"2024-02-14T00:54:40.443641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_applprev_1.group_by('case_id').mean(), 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T00:55:58.075973Z","iopub.execute_input":"2024-02-14T00:55:58.076420Z","iopub.status.idle":"2024-02-14T00:56:10.017251Z","shell.execute_reply.started":"2024-02-14T00:55:58.076383Z","shell.execute_reply":"2024-02-14T00:56:10.016292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.violinplot(\n    (\n        df_train_base.lazy()\n        .join(df_train_applprev_1.group_by('case_id').agg(pl.len().alias('applprev_1_count')), on = 'case_id', how = 'left')\n        .select(pl.col('applprev_1_count').fill_null(0), 'target')\n        .collect().to_pandas()\n    ),\n    x = 'target', y = 'applprev_1_count',\n    orient = 'v', order = [0, 1],\n);","metadata":{"execution":{"iopub.status.busy":"2024-02-14T06:13:18.125484Z","iopub.execute_input":"2024-02-14T06:13:18.125815Z","iopub.status.idle":"2024-02-14T06:13:22.380752Z","shell.execute_reply.started":"2024-02-14T06:13:18.125792Z","shell.execute_reply":"2024-02-14T06:13:22.379305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_rows(-1)\n    for f in glob.iglob('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_applprev_1_*.parquet'):\n        display(\n            pl.scan_parquet(f)\n            .join(df_train_base.lazy(), on = 'case_id', how = 'left')\n            .select(pl.col('WEEK_NUM').unique())\n            .collect()\n        )","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:02:42.391185Z","iopub.execute_input":"2024-03-13T01:02:42.391625Z","iopub.status.idle":"2024-03-13T01:02:43.255979Z","shell.execute_reply.started":"2024-03-13T01:02:42.391592Z","shell.execute_reply":"2024-03-13T01:02:43.254579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# other_1","metadata":{}},{"cell_type":"code","source":"df_train_other_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_other_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:15:05.283294Z","iopub.execute_input":"2024-02-26T01:15:05.283740Z","iopub.status.idle":"2024-02-26T01:15:05.298225Z","shell.execute_reply.started":"2024-02-26T01:15:05.283708Z","shell.execute_reply":"2024-02-26T01:15:05.296979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_other_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:40:41.376986Z","iopub.execute_input":"2024-02-13T01:40:41.378042Z","iopub.status.idle":"2024-02-13T01:40:41.388312Z","shell.execute_reply.started":"2024-02-13T01:40:41.378003Z","shell.execute_reply":"2024-02-13T01:40:41.386930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_other_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:40:42.160171Z","iopub.execute_input":"2024-02-13T01:40:42.161292Z","iopub.status.idle":"2024-02-13T01:40:42.184900Z","shell.execute_reply.started":"2024-02-13T01:40:42.161250Z","shell.execute_reply":"2024-02-13T01:40:42.183647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_other_1.collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:40:42.920958Z","iopub.execute_input":"2024-02-13T01:40:42.921586Z","iopub.status.idle":"2024-02-13T01:40:42.946152Z","shell.execute_reply.started":"2024-02-13T01:40:42.921553Z","shell.execute_reply":"2024-02-13T01:40:42.944861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_other_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:15:45.912024Z","iopub.execute_input":"2024-02-26T01:15:45.912478Z","iopub.status.idle":"2024-02-26T01:15:46.298587Z","shell.execute_reply.started":"2024-02-26T01:15:45.912443Z","shell.execute_reply":"2024-02-26T01:15:46.297674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_other_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:41:11.543766Z","iopub.execute_input":"2024-02-13T01:41:11.544287Z","iopub.status.idle":"2024-02-13T01:41:15.940625Z","shell.execute_reply.started":"2024-02-13T01:41:11.544241Z","shell.execute_reply":"2024-02-13T01:41:15.939590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_other_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:35:02.355653Z","iopub.execute_input":"2024-02-13T04:35:02.356796Z","iopub.status.idle":"2024-02-13T04:35:02.782983Z","shell.execute_reply.started":"2024-02-13T04:35:02.356753Z","shell.execute_reply":"2024-02-13T04:35:02.781747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_other_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T00:57:00.259387Z","iopub.execute_input":"2024-02-14T00:57:00.259790Z","iopub.status.idle":"2024-02-14T00:57:14.929811Z","shell.execute_reply.started":"2024-02-14T00:57:00.259759Z","shell.execute_reply":"2024-02-14T00:57:14.928853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# tax_registry_a_1","metadata":{}},{"cell_type":"code","source":"df_train_tax_registry_a_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_tax_registry_a_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:15:57.575916Z","iopub.execute_input":"2024-02-26T01:15:57.577640Z","iopub.status.idle":"2024-02-26T01:15:57.598776Z","shell.execute_reply.started":"2024-02-26T01:15:57.577586Z","shell.execute_reply":"2024-02-26T01:15:57.597113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_tax_registry_a_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:41:23.330080Z","iopub.execute_input":"2024-02-13T01:41:23.330538Z","iopub.status.idle":"2024-02-13T01:41:23.342785Z","shell.execute_reply.started":"2024-02-13T01:41:23.330503Z","shell.execute_reply":"2024-02-13T01:41:23.341531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_a_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:41:24.042087Z","iopub.execute_input":"2024-02-13T01:41:24.042886Z","iopub.status.idle":"2024-02-13T01:41:24.089240Z","shell.execute_reply.started":"2024-02-13T01:41:24.042847Z","shell.execute_reply":"2024-02-13T01:41:24.087404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_a_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:41:24.777909Z","iopub.execute_input":"2024-02-13T01:41:24.778327Z","iopub.status.idle":"2024-02-13T01:41:27.500704Z","shell.execute_reply.started":"2024-02-13T01:41:24.778290Z","shell.execute_reply":"2024-02-13T01:41:27.499528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_a_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:41:36.178124Z","iopub.execute_input":"2024-02-13T01:41:36.178523Z","iopub.status.idle":"2024-02-13T01:41:36.602404Z","shell.execute_reply.started":"2024-02-13T01:41:36.178493Z","shell.execute_reply":"2024-02-13T01:41:36.601166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_tax_registry_a_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:17:17.390123Z","iopub.execute_input":"2024-02-26T01:17:17.390903Z","iopub.status.idle":"2024-02-26T01:17:18.420481Z","shell.execute_reply.started":"2024-02-26T01:17:17.390849Z","shell.execute_reply":"2024-02-26T01:17:18.419304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_tax_registry_a_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:42:25.408844Z","iopub.execute_input":"2024-02-13T01:42:25.410015Z","iopub.status.idle":"2024-02-13T01:42:40.756528Z","shell.execute_reply.started":"2024-02-13T01:42:25.409968Z","shell.execute_reply":"2024-02-13T01:42:40.755360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_tax_registry_a_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:42:40.758651Z","iopub.execute_input":"2024-02-13T01:42:40.759828Z","iopub.status.idle":"2024-02-13T01:42:42.677004Z","shell.execute_reply.started":"2024-02-13T01:42:40.759780Z","shell.execute_reply":"2024-02-13T01:42:42.675779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_tax_registry_a_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T00:58:34.067382Z","iopub.execute_input":"2024-02-14T00:58:34.067904Z","iopub.status.idle":"2024-02-14T00:58:38.243045Z","shell.execute_reply.started":"2024-02-14T00:58:34.067866Z","shell.execute_reply":"2024-02-14T00:58:38.241712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# tax_registry_b_1","metadata":{}},{"cell_type":"code","source":"df_train_tax_registry_b_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_tax_registry_b_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:17:46.208727Z","iopub.execute_input":"2024-02-26T01:17:46.209177Z","iopub.status.idle":"2024-02-26T01:17:46.223814Z","shell.execute_reply.started":"2024-02-26T01:17:46.209142Z","shell.execute_reply":"2024-02-26T01:17:46.222802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_tax_registry_b_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:43:18.233646Z","iopub.execute_input":"2024-02-13T01:43:18.234269Z","iopub.status.idle":"2024-02-13T01:43:18.244263Z","shell.execute_reply.started":"2024-02-13T01:43:18.234236Z","shell.execute_reply":"2024-02-13T01:43:18.242966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_b_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:43:20.709670Z","iopub.execute_input":"2024-02-13T01:43:20.710325Z","iopub.status.idle":"2024-02-13T01:43:20.759222Z","shell.execute_reply.started":"2024-02-13T01:43:20.710291Z","shell.execute_reply":"2024-02-13T01:43:20.757998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_b_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:43:21.405208Z","iopub.execute_input":"2024-02-13T01:43:21.405854Z","iopub.status.idle":"2024-02-13T01:43:22.355399Z","shell.execute_reply.started":"2024-02-13T01:43:21.405821Z","shell.execute_reply":"2024-02-13T01:43:22.354322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_b_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:43:22.357276Z","iopub.execute_input":"2024-02-13T01:43:22.358007Z","iopub.status.idle":"2024-02-13T01:43:22.498179Z","shell.execute_reply.started":"2024-02-13T01:43:22.357971Z","shell.execute_reply":"2024-02-13T01:43:22.497196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_tax_registry_b_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:18:23.479243Z","iopub.execute_input":"2024-02-26T01:18:23.479651Z","iopub.status.idle":"2024-02-26T01:18:23.985607Z","shell.execute_reply.started":"2024-02-26T01:18:23.479620Z","shell.execute_reply":"2024-02-26T01:18:23.984473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_tax_registry_b_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:43:43.266723Z","iopub.execute_input":"2024-02-13T01:43:43.267112Z","iopub.status.idle":"2024-02-13T01:43:49.106046Z","shell.execute_reply.started":"2024-02-13T01:43:43.267083Z","shell.execute_reply":"2024-02-13T01:43:49.104881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_tax_registry_b_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:43:51.425434Z","iopub.execute_input":"2024-02-13T01:43:51.425821Z","iopub.status.idle":"2024-02-13T01:43:52.748138Z","shell.execute_reply.started":"2024-02-13T01:43:51.425792Z","shell.execute_reply":"2024-02-13T01:43:52.746829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_tax_registry_b_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T00:59:29.852409Z","iopub.execute_input":"2024-02-14T00:59:29.852859Z","iopub.status.idle":"2024-02-14T00:59:33.357472Z","shell.execute_reply.started":"2024-02-14T00:59:29.852825Z","shell.execute_reply":"2024-02-14T00:59:33.356479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# tax_registry_c_1","metadata":{}},{"cell_type":"code","source":"df_train_tax_registry_c_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_tax_registry_c_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:18:30.454291Z","iopub.execute_input":"2024-02-26T01:18:30.455307Z","iopub.status.idle":"2024-02-26T01:18:30.469914Z","shell.execute_reply.started":"2024-02-26T01:18:30.455238Z","shell.execute_reply":"2024-02-26T01:18:30.468532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_tax_registry_c_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:44:08.514974Z","iopub.execute_input":"2024-02-13T01:44:08.515440Z","iopub.status.idle":"2024-02-13T01:44:08.525957Z","shell.execute_reply.started":"2024-02-13T01:44:08.515404Z","shell.execute_reply":"2024-02-13T01:44:08.524706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_c_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:44:08.991365Z","iopub.execute_input":"2024-02-13T01:44:08.991811Z","iopub.status.idle":"2024-02-13T01:44:09.067307Z","shell.execute_reply.started":"2024-02-13T01:44:08.991775Z","shell.execute_reply":"2024-02-13T01:44:09.066448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_c_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:44:09.469796Z","iopub.execute_input":"2024-02-13T01:44:09.470451Z","iopub.status.idle":"2024-02-13T01:44:12.318732Z","shell.execute_reply.started":"2024-02-13T01:44:09.470417Z","shell.execute_reply":"2024-02-13T01:44:12.317508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_tax_registry_c_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:44:12.322850Z","iopub.execute_input":"2024-02-13T01:44:12.323603Z","iopub.status.idle":"2024-02-13T01:44:12.784641Z","shell.execute_reply.started":"2024-02-13T01:44:12.323567Z","shell.execute_reply":"2024-02-13T01:44:12.783354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_tax_registry_c_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:19:19.408353Z","iopub.execute_input":"2024-02-26T01:19:19.408776Z","iopub.status.idle":"2024-02-26T01:19:20.305693Z","shell.execute_reply.started":"2024-02-26T01:19:19.408743Z","shell.execute_reply":"2024-02-26T01:19:20.304465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_tax_registry_c_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:44:33.578653Z","iopub.execute_input":"2024-02-13T01:44:33.579080Z","iopub.status.idle":"2024-02-13T01:44:49.163319Z","shell.execute_reply.started":"2024-02-13T01:44:33.579049Z","shell.execute_reply":"2024-02-13T01:44:49.162040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_tax_registry_c_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:44:49.165455Z","iopub.execute_input":"2024-02-13T01:44:49.166139Z","iopub.status.idle":"2024-02-13T01:44:51.119238Z","shell.execute_reply.started":"2024-02-13T01:44:49.166099Z","shell.execute_reply":"2024-02-13T01:44:51.117900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_tax_registry_c_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:00:05.800709Z","iopub.execute_input":"2024-02-14T01:00:05.801096Z","iopub.status.idle":"2024-02-14T01:00:10.409911Z","shell.execute_reply.started":"2024-02-14T01:00:05.801067Z","shell.execute_reply":"2024-02-14T01:00:10.408511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- All of `tax_registry_[a/b/c]_1` seem to have essentially the same columns but with different names.\n- Names are high-cardinality categorical variables.\n  - Single majority and many minorities.","metadata":{}},{"cell_type":"markdown","source":"# credit_bureau_a_1","metadata":{}},{"cell_type":"code","source":"df_train_credit_bureau_a_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_credit_bureau_a_1_*.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:04:08.440117Z","iopub.execute_input":"2024-03-13T01:04:08.441504Z","iopub.status.idle":"2024-03-13T01:04:08.465833Z","shell.execute_reply.started":"2024-03-13T01:04:08.441459Z","shell.execute_reply":"2024-03-13T01:04:08.463978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    cfg.set_tbl_rows(-1)\n    display(get_feature_definitions(df_train_credit_bureau_a_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:37:07.016760Z","iopub.execute_input":"2024-02-14T07:37:07.017172Z","iopub.status.idle":"2024-02-14T07:37:07.035982Z","shell.execute_reply.started":"2024-02-14T07:37:07.017143Z","shell.execute_reply":"2024-02-14T07:37:07.035117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_cols(-1)\n    display(df_train_credit_bureau_a_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:37:20.161757Z","iopub.execute_input":"2024-02-14T07:37:20.162144Z","iopub.status.idle":"2024-02-14T07:37:20.423983Z","shell.execute_reply.started":"2024-02-14T07:37:20.162115Z","shell.execute_reply":"2024-02-14T07:37:20.422515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_cols(-1)\n    display(df_train_credit_bureau_a_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:37:24.385118Z","iopub.execute_input":"2024-02-14T07:37:24.385528Z","iopub.status.idle":"2024-02-14T07:38:10.424591Z","shell.execute_reply.started":"2024-02-14T07:37:24.385469Z","shell.execute_reply":"2024-02-14T07:38:10.422989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The max of `dpdmax_757P` is `657458 days ~ 1801 years`!?!?","metadata":{}},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_cols(-1)\n    display(df_train_credit_bureau_a_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:47:11.675149Z","iopub.execute_input":"2024-02-13T01:47:11.675634Z","iopub.status.idle":"2024-02-13T01:47:15.503923Z","shell.execute_reply.started":"2024-02-13T01:47:11.675589Z","shell.execute_reply":"2024-02-13T01:47:15.502436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_credit_bureau_a_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:21:26.905286Z","iopub.execute_input":"2024-02-26T01:21:26.905703Z","iopub.status.idle":"2024-02-26T01:21:29.190961Z","shell.execute_reply.started":"2024-02-26T01:21:26.905672Z","shell.execute_reply":"2024-02-26T01:21:29.189284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_a_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:47:15.538218Z","iopub.execute_input":"2024-02-13T01:47:15.538732Z","iopub.status.idle":"2024-02-13T01:55:37.830315Z","shell.execute_reply.started":"2024-02-13T01:47:15.538685Z","shell.execute_reply":"2024-02-13T01:55:37.828750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_a_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:38:10.426699Z","iopub.execute_input":"2024-02-14T07:38:10.427161Z","iopub.status.idle":"2024-02-14T07:38:15.768257Z","shell.execute_reply.started":"2024-02-14T07:38:10.427122Z","shell.execute_reply":"2024-02-14T07:38:15.767087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_credit_bureau_a_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:56:41.578045Z","iopub.execute_input":"2024-02-13T01:56:41.578755Z","iopub.status.idle":"2024-02-13T01:57:29.254933Z","shell.execute_reply.started":"2024-02-13T01:56:41.578719Z","shell.execute_reply":"2024-02-13T01:57:29.253676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_credit_bureau_a_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:36:27.975907Z","iopub.execute_input":"2024-02-13T04:36:27.976339Z","iopub.status.idle":"2024-02-13T04:36:35.589007Z","shell.execute_reply.started":"2024-02-13T04:36:27.976309Z","shell.execute_reply":"2024-02-13T04:36:35.587889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_credit_bureau_a_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:38:04.667416Z","iopub.execute_input":"2024-02-13T04:38:04.667779Z","iopub.status.idle":"2024-02-13T04:38:05.357932Z","shell.execute_reply.started":"2024-02-13T04:38:04.667751Z","shell.execute_reply":"2024-02-13T04:38:05.356839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_a_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:09:16.655656Z","iopub.execute_input":"2024-02-13T05:09:16.656123Z","iopub.status.idle":"2024-02-13T05:09:32.542672Z","shell.execute_reply.started":"2024-02-13T05:09:16.656091Z","shell.execute_reply":"2024-02-13T05:09:32.541641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_a_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:09:32.544623Z","iopub.execute_input":"2024-02-13T05:09:32.545511Z","iopub.status.idle":"2024-02-13T05:09:33.970351Z","shell.execute_reply.started":"2024-02-13T05:09:32.545477Z","shell.execute_reply":"2024-02-13T05:09:33.969234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_a_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:09:48.352672Z","iopub.execute_input":"2024-02-13T05:09:48.353099Z","iopub.status.idle":"2024-02-13T05:09:55.774995Z","shell.execute_reply.started":"2024-02-13T05:09:48.353066Z","shell.execute_reply":"2024-02-13T05:09:55.773832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_a_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:00:52.601919Z","iopub.execute_input":"2024-02-14T01:00:52.602351Z","iopub.status.idle":"2024-02-14T01:03:08.555915Z","shell.execute_reply.started":"2024-02-14T01:00:52.602316Z","shell.execute_reply":"2024-02-14T01:03:08.554547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_a_1.group_by('case_id').mean(), 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:04:05.126271Z","iopub.execute_input":"2024-02-14T01:04:05.126676Z","iopub.status.idle":"2024-02-14T01:04:19.015309Z","shell.execute_reply.started":"2024-02-14T01:04:05.126645Z","shell.execute_reply":"2024-02-14T01:04:19.013922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_rows(-1)\n    for f in glob.iglob('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_credit_bureau_a_1_*.parquet'):\n        display(\n            pl.scan_parquet(f)\n            .join(df_train_base.lazy(), on = 'case_id', how = 'left')\n            .select(pl.col('WEEK_NUM').unique())\n            .collect()\n        )","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:05:03.636452Z","iopub.execute_input":"2024-03-13T01:05:03.636996Z","iopub.status.idle":"2024-03-13T01:05:05.753702Z","shell.execute_reply.started":"2024-03-13T01:05:03.636950Z","shell.execute_reply":"2024-03-13T01:05:05.751974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# credit_bureau_b_1","metadata":{}},{"cell_type":"code","source":"df_train_credit_bureau_b_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_credit_bureau_b_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:21:40.443386Z","iopub.execute_input":"2024-02-26T01:21:40.444520Z","iopub.status.idle":"2024-02-26T01:21:40.468532Z","shell.execute_reply.started":"2024-02-26T01:21:40.444454Z","shell.execute_reply":"2024-02-26T01:21:40.467100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    cfg.set_tbl_rows(-1)\n    display(get_feature_definitions(df_train_credit_bureau_b_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:42:25.891244Z","iopub.execute_input":"2024-02-14T07:42:25.892515Z","iopub.status.idle":"2024-02-14T07:42:25.905703Z","shell.execute_reply.started":"2024-02-14T07:42:25.892449Z","shell.execute_reply":"2024-02-14T07:42:25.904565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_credit_bureau_b_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:42:26.555429Z","iopub.execute_input":"2024-02-14T07:42:26.555844Z","iopub.status.idle":"2024-02-14T07:42:26.603281Z","shell.execute_reply.started":"2024-02-14T07:42:26.555813Z","shell.execute_reply":"2024-02-14T07:42:26.602336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_credit_bureau_b_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:42:27.304082Z","iopub.execute_input":"2024-02-14T07:42:27.305018Z","iopub.status.idle":"2024-02-14T07:42:27.521266Z","shell.execute_reply.started":"2024-02-14T07:42:27.304983Z","shell.execute_reply":"2024-02-14T07:42:27.520461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The max values of `dpd_500P`, `dpd_733P`, `dpdmax_851P`, and `pmtdaysoverdue_1135P` are greater than `600000 days ~ 1643 years`.","metadata":{}},{"cell_type":"code","source":"display(df_train_credit_bureau_b_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:58:06.652091Z","iopub.execute_input":"2024-02-13T01:58:06.652829Z","iopub.status.idle":"2024-02-13T01:58:06.678242Z","shell.execute_reply.started":"2024-02-13T01:58:06.652795Z","shell.execute_reply":"2024-02-13T01:58:06.676882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_credit_bureau_b_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:22:17.665392Z","iopub.execute_input":"2024-02-26T01:22:17.665832Z","iopub.status.idle":"2024-02-26T01:22:18.033185Z","shell.execute_reply.started":"2024-02-26T01:22:17.665799Z","shell.execute_reply":"2024-02-26T01:22:18.032047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_b_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:58:38.683413Z","iopub.execute_input":"2024-02-13T01:58:38.683847Z","iopub.status.idle":"2024-02-13T01:58:52.695955Z","shell.execute_reply.started":"2024-02-13T01:58:38.683815Z","shell.execute_reply":"2024-02-13T01:58:52.694666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_b_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:58:54.825966Z","iopub.execute_input":"2024-02-13T01:58:54.826398Z","iopub.status.idle":"2024-02-13T01:59:06.752132Z","shell.execute_reply.started":"2024-02-13T01:58:54.826365Z","shell.execute_reply":"2024-02-13T01:59:06.750882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_credit_bureau_b_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T01:59:06.754490Z","iopub.execute_input":"2024-02-13T01:59:06.755368Z","iopub.status.idle":"2024-02-13T01:59:12.132710Z","shell.execute_reply.started":"2024-02-13T01:59:06.755305Z","shell.execute_reply":"2024-02-13T01:59:12.131554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_credit_bureau_b_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:38:34.037266Z","iopub.execute_input":"2024-02-13T04:38:34.038115Z","iopub.status.idle":"2024-02-13T04:38:35.401181Z","shell.execute_reply.started":"2024-02-13T04:38:34.038079Z","shell.execute_reply":"2024-02-13T04:38:35.400064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_credit_bureau_b_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:38:55.513899Z","iopub.execute_input":"2024-02-13T04:38:55.514383Z","iopub.status.idle":"2024-02-13T04:38:55.863975Z","shell.execute_reply.started":"2024-02-13T04:38:55.514345Z","shell.execute_reply":"2024-02-13T04:38:55.862674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_b_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:10:15.418282Z","iopub.execute_input":"2024-02-13T05:10:15.418702Z","iopub.status.idle":"2024-02-13T05:10:18.289240Z","shell.execute_reply.started":"2024-02-13T05:10:15.418671Z","shell.execute_reply":"2024-02-13T05:10:18.288034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_b_1, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:10:23.961099Z","iopub.execute_input":"2024-02-13T05:10:23.962054Z","iopub.status.idle":"2024-02-13T05:10:24.725462Z","shell.execute_reply.started":"2024-02-13T05:10:23.962010Z","shell.execute_reply":"2024-02-13T05:10:24.724373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_b_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:10:30.416903Z","iopub.execute_input":"2024-02-13T05:10:30.417311Z","iopub.status.idle":"2024-02-13T05:10:31.291084Z","shell.execute_reply.started":"2024-02-13T05:10:30.417281Z","shell.execute_reply":"2024-02-13T05:10:31.289986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_b_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:06:06.898358Z","iopub.execute_input":"2024-02-14T01:06:06.898798Z","iopub.status.idle":"2024-02-14T01:06:53.083815Z","shell.execute_reply.started":"2024-02-14T01:06:06.898764Z","shell.execute_reply":"2024-02-14T01:06:53.078805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_b_1.group_by('case_id').mean(), 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:07:45.099770Z","iopub.execute_input":"2024-02-14T01:07:45.100168Z","iopub.status.idle":"2024-02-14T01:07:56.754440Z","shell.execute_reply.started":"2024-02-14T01:07:45.100138Z","shell.execute_reply":"2024-02-14T01:07:56.753064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# deposit_1","metadata":{}},{"cell_type":"code","source":"df_train_deposit_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_deposit_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:22:31.373917Z","iopub.execute_input":"2024-02-26T01:22:31.374595Z","iopub.status.idle":"2024-02-26T01:22:31.390892Z","shell.execute_reply.started":"2024-02-26T01:22:31.374551Z","shell.execute_reply":"2024-02-26T01:22:31.389625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_deposit_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:00:16.213014Z","iopub.execute_input":"2024-02-13T02:00:16.213475Z","iopub.status.idle":"2024-02-13T02:00:16.224774Z","shell.execute_reply.started":"2024-02-13T02:00:16.213439Z","shell.execute_reply":"2024-02-13T02:00:16.223236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_deposit_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:00:16.768007Z","iopub.execute_input":"2024-02-13T02:00:16.768474Z","iopub.status.idle":"2024-02-13T02:00:16.798477Z","shell.execute_reply.started":"2024-02-13T02:00:16.768439Z","shell.execute_reply":"2024-02-13T02:00:16.797426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_deposit_1.collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:00:17.565744Z","iopub.execute_input":"2024-02-13T02:00:17.566918Z","iopub.status.idle":"2024-02-13T02:00:17.698134Z","shell.execute_reply.started":"2024-02-13T02:00:17.566872Z","shell.execute_reply":"2024-02-13T02:00:17.696637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_deposit_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:23:18.604382Z","iopub.execute_input":"2024-02-26T01:23:18.604882Z","iopub.status.idle":"2024-02-26T01:23:19.056888Z","shell.execute_reply.started":"2024-02-26T01:23:18.604844Z","shell.execute_reply":"2024-02-26T01:23:19.055878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_deposit_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:00:43.787800Z","iopub.execute_input":"2024-02-13T02:00:43.788266Z","iopub.status.idle":"2024-02-13T02:00:45.142990Z","shell.execute_reply.started":"2024-02-13T02:00:43.788229Z","shell.execute_reply":"2024-02-13T02:00:45.141460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_deposit_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:13:48.315131Z","iopub.execute_input":"2024-02-14T01:13:48.315587Z","iopub.status.idle":"2024-02-14T01:13:51.714903Z","shell.execute_reply.started":"2024-02-14T01:13:48.315549Z","shell.execute_reply":"2024-02-14T01:13:51.713499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# person_1","metadata":{}},{"cell_type":"code","source":"df_train_person_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_person_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:23:43.312612Z","iopub.execute_input":"2024-02-26T01:23:43.313068Z","iopub.status.idle":"2024-02-26T01:23:43.328584Z","shell.execute_reply.started":"2024-02-26T01:23:43.313033Z","shell.execute_reply":"2024-02-26T01:23:43.327235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    cfg.set_tbl_rows(-1)\n    display(get_feature_definitions(df_train_person_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:41:01.034889Z","iopub.execute_input":"2024-02-14T01:41:01.035721Z","iopub.status.idle":"2024-02-14T01:41:01.051434Z","shell.execute_reply.started":"2024-02-14T01:41:01.035666Z","shell.execute_reply":"2024-02-14T01:41:01.050035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:41:13.913384Z","iopub.execute_input":"2024-02-14T01:41:13.913865Z","iopub.status.idle":"2024-02-14T01:41:14.045529Z","shell.execute_reply.started":"2024-02-14T01:41:13.913830Z","shell.execute_reply":"2024-02-14T01:41:14.044517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_1.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:41:36.563451Z","iopub.execute_input":"2024-02-14T01:41:36.563856Z","iopub.status.idle":"2024-02-14T01:41:38.620238Z","shell.execute_reply.started":"2024-02-14T01:41:36.563825Z","shell.execute_reply":"2024-02-14T01:41:38.618917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_1.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:43:26.638146Z","iopub.execute_input":"2024-02-14T01:43:26.638661Z","iopub.status.idle":"2024-02-14T01:43:27.966281Z","shell.execute_reply.started":"2024-02-14T01:43:26.638603Z","shell.execute_reply":"2024-02-14T01:43:27.964976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_person_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:24:21.213390Z","iopub.execute_input":"2024-02-26T01:24:21.213897Z","iopub.status.idle":"2024-02-26T01:24:23.124832Z","shell.execute_reply.started":"2024-02-26T01:24:21.213860Z","shell.execute_reply":"2024-02-26T01:24:23.123840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_person_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:43:56.014329Z","iopub.execute_input":"2024-02-14T01:43:56.014811Z","iopub.status.idle":"2024-02-14T01:44:01.187772Z","shell.execute_reply.started":"2024-02-14T01:43:56.014727Z","shell.execute_reply":"2024-02-14T01:44:01.186568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_person_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:44:10.518960Z","iopub.execute_input":"2024-02-14T01:44:10.519370Z","iopub.status.idle":"2024-02-14T01:44:19.274390Z","shell.execute_reply.started":"2024-02-14T01:44:10.519331Z","shell.execute_reply":"2024-02-14T01:44:19.272093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Addresses and zip codes are high-cardinality variables.\n  - One majority and many minorities.","metadata":{}},{"cell_type":"code","source":"plot_null(df_train_person_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:11:31.850515Z","iopub.execute_input":"2024-02-13T05:11:31.850897Z","iopub.status.idle":"2024-02-13T05:11:32.708701Z","shell.execute_reply.started":"2024-02-13T05:11:31.850870Z","shell.execute_reply":"2024-02-13T05:11:32.707613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_person_1, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:11:46.005007Z","iopub.execute_input":"2024-02-13T05:11:46.005399Z","iopub.status.idle":"2024-02-13T05:11:49.922108Z","shell.execute_reply.started":"2024-02-13T05:11:46.005369Z","shell.execute_reply":"2024-02-13T05:11:49.920665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_person_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:15:21.797694Z","iopub.execute_input":"2024-02-14T01:15:21.798091Z","iopub.status.idle":"2024-02-14T01:15:28.596081Z","shell.execute_reply.started":"2024-02-14T01:15:21.798061Z","shell.execute_reply":"2024-02-14T01:15:28.594599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> In case `num_group1` or `num_group2` stands for person index (this is clear with predictor definitions) the zero index has special meaning. When `num_groupN=0` it is the applicant (the person who applied for a loan).","metadata":{}},{"cell_type":"code","source":"display(df_train_person_1.filter(pl.col('num_group1') == 0).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:45:50.441282Z","iopub.execute_input":"2024-02-14T01:45:50.441746Z","iopub.status.idle":"2024-02-14T01:45:54.020608Z","shell.execute_reply.started":"2024-02-14T01:45:50.441700Z","shell.execute_reply":"2024-02-14T01:45:54.018955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_1.filter(pl.col('num_group1') != 0).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:46:57.075231Z","iopub.execute_input":"2024-02-14T01:46:57.075769Z","iopub.status.idle":"2024-02-14T01:46:59.288861Z","shell.execute_reply.started":"2024-02-14T01:46:57.075725Z","shell.execute_reply":"2024-02-14T01:46:59.287882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Some columns missing all values for one of `num_group1 == 0` and `num_group1 != 0`","metadata":{}},{"cell_type":"markdown","source":"# debitcard_1","metadata":{}},{"cell_type":"code","source":"df_train_debitcard_1 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_debitcard_1.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:24:48.947557Z","iopub.execute_input":"2024-02-26T01:24:48.948603Z","iopub.status.idle":"2024-02-26T01:24:48.964840Z","shell.execute_reply.started":"2024-02-26T01:24:48.948565Z","shell.execute_reply":"2024-02-26T01:24:48.963256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_debitcard_1.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:04:42.395141Z","iopub.execute_input":"2024-02-13T02:04:42.395625Z","iopub.status.idle":"2024-02-13T02:04:42.405931Z","shell.execute_reply.started":"2024-02-13T02:04:42.395588Z","shell.execute_reply":"2024-02-13T02:04:42.404522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_debitcard_1.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:04:43.114215Z","iopub.execute_input":"2024-02-13T02:04:43.115761Z","iopub.status.idle":"2024-02-13T02:04:43.148557Z","shell.execute_reply.started":"2024-02-13T02:04:43.115713Z","shell.execute_reply":"2024-02-13T02:04:43.147042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_debitcard_1.collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:04:48.544245Z","iopub.execute_input":"2024-02-13T02:04:48.545460Z","iopub.status.idle":"2024-02-13T02:04:48.679257Z","shell.execute_reply.started":"2024-02-13T02:04:48.545412Z","shell.execute_reply":"2024-02-13T02:04:48.677991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_debitcard_1.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:25:25.911481Z","iopub.execute_input":"2024-02-26T01:25:25.912217Z","iopub.status.idle":"2024-02-26T01:25:26.376057Z","shell.execute_reply.started":"2024-02-26T01:25:25.912159Z","shell.execute_reply":"2024-02-26T01:25:26.374590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_debitcard_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:05:06.810775Z","iopub.execute_input":"2024-02-13T02:05:06.811211Z","iopub.status.idle":"2024-02-13T02:05:09.235213Z","shell.execute_reply.started":"2024-02-13T02:05:06.811170Z","shell.execute_reply":"2024-02-13T02:05:09.233742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_debitcard_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:40:11.308193Z","iopub.execute_input":"2024-02-13T04:40:11.308672Z","iopub.status.idle":"2024-02-13T04:40:11.603068Z","shell.execute_reply.started":"2024-02-13T04:40:11.308629Z","shell.execute_reply":"2024-02-13T04:40:11.601975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_debitcard_1, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:12:14.028344Z","iopub.execute_input":"2024-02-13T05:12:14.029056Z","iopub.status.idle":"2024-02-13T05:12:14.749497Z","shell.execute_reply.started":"2024-02-13T05:12:14.029018Z","shell.execute_reply":"2024-02-13T05:12:14.748544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_debitcard_1.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:16:10.692207Z","iopub.execute_input":"2024-02-14T01:16:10.692649Z","iopub.status.idle":"2024-02-14T01:16:19.326067Z","shell.execute_reply.started":"2024-02-14T01:16:10.692602Z","shell.execute_reply":"2024-02-14T01:16:19.324821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# applprev_2","metadata":{}},{"cell_type":"code","source":"df_train_applprev_2 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_applprev_2.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:25:33.442597Z","iopub.execute_input":"2024-02-26T01:25:33.443378Z","iopub.status.idle":"2024-02-26T01:25:33.459376Z","shell.execute_reply.started":"2024-02-26T01:25:33.443251Z","shell.execute_reply":"2024-02-26T01:25:33.457934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_applprev_2.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:05:31.714083Z","iopub.execute_input":"2024-02-13T02:05:31.714503Z","iopub.status.idle":"2024-02-13T02:05:31.724534Z","shell.execute_reply.started":"2024-02-13T02:05:31.714470Z","shell.execute_reply":"2024-02-13T02:05:31.723301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_applprev_2.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:05:32.179942Z","iopub.execute_input":"2024-02-13T02:05:32.181155Z","iopub.status.idle":"2024-02-13T02:05:32.227657Z","shell.execute_reply.started":"2024-02-13T02:05:32.181090Z","shell.execute_reply":"2024-02-13T02:05:32.226593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_applprev_2.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:05:32.839398Z","iopub.execute_input":"2024-02-13T02:05:32.840120Z","iopub.status.idle":"2024-02-13T02:05:34.213066Z","shell.execute_reply.started":"2024-02-13T02:05:32.840054Z","shell.execute_reply":"2024-02-13T02:05:34.211825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_applprev_2.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:35:36.983473Z","iopub.execute_input":"2024-02-26T01:35:36.984603Z","iopub.status.idle":"2024-02-26T01:35:39.948871Z","shell.execute_reply.started":"2024-02-26T01:35:36.984560Z","shell.execute_reply":"2024-02-26T01:35:39.947224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_applprev_2.group_by('case_id', 'num_group1')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:35:56.547703Z","iopub.execute_input":"2024-02-26T01:35:56.548174Z","iopub.status.idle":"2024-02-26T01:36:04.216804Z","shell.execute_reply.started":"2024-02-26T01:35:56.548142Z","shell.execute_reply":"2024-02-26T01:36:04.215277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_applprev_2, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:05:48.380556Z","iopub.execute_input":"2024-02-13T02:05:48.380955Z","iopub.status.idle":"2024-02-13T02:05:51.882091Z","shell.execute_reply.started":"2024-02-13T02:05:48.380926Z","shell.execute_reply":"2024-02-13T02:05:51.880388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_applprev_2, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:12:53.142788Z","iopub.execute_input":"2024-02-13T05:12:53.143247Z","iopub.status.idle":"2024-02-13T05:12:54.588141Z","shell.execute_reply.started":"2024-02-13T05:12:53.143213Z","shell.execute_reply":"2024-02-13T05:12:54.587191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# person_2","metadata":{}},{"cell_type":"code","source":"df_train_person_2 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_person_2.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:28:18.912554Z","iopub.execute_input":"2024-02-26T01:28:18.913087Z","iopub.status.idle":"2024-02-26T01:28:18.938851Z","shell.execute_reply.started":"2024-02-26T01:28:18.913047Z","shell.execute_reply":"2024-02-26T01:28:18.937585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_person_2.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:54:37.235902Z","iopub.execute_input":"2024-02-14T01:54:37.236519Z","iopub.status.idle":"2024-02-14T01:54:37.247481Z","shell.execute_reply.started":"2024-02-14T01:54:37.236486Z","shell.execute_reply":"2024-02-14T01:54:37.246006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_2.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:54:38.314721Z","iopub.execute_input":"2024-02-14T01:54:38.315171Z","iopub.status.idle":"2024-02-14T01:54:38.354053Z","shell.execute_reply.started":"2024-02-14T01:54:38.315133Z","shell.execute_reply":"2024-02-14T01:54:38.353019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_2.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:54:41.051358Z","iopub.execute_input":"2024-02-14T01:54:41.052143Z","iopub.status.idle":"2024-02-14T01:54:41.314839Z","shell.execute_reply.started":"2024-02-14T01:54:41.052096Z","shell.execute_reply":"2024-02-14T01:54:41.313453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_2.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:54:58.608091Z","iopub.execute_input":"2024-02-14T01:54:58.608493Z","iopub.status.idle":"2024-02-14T01:54:58.850877Z","shell.execute_reply.started":"2024-02-14T01:54:58.608462Z","shell.execute_reply":"2024-02-14T01:54:58.849857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_person_2.group_by('case_id', 'num_group1')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:29:05.027953Z","iopub.execute_input":"2024-02-26T01:29:05.028351Z","iopub.status.idle":"2024-02-26T01:29:06.803956Z","shell.execute_reply.started":"2024-02-26T01:29:05.028318Z","shell.execute_reply":"2024-02-26T01:29:06.802651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_person_2, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:55:01.910341Z","iopub.execute_input":"2024-02-14T01:55:01.911168Z","iopub.status.idle":"2024-02-14T01:55:08.246305Z","shell.execute_reply.started":"2024-02-14T01:55:01.911123Z","shell.execute_reply":"2024-02-14T01:55:08.244793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_person_2, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:55:08.248504Z","iopub.execute_input":"2024-02-14T01:55:08.249596Z","iopub.status.idle":"2024-02-14T01:55:10.413231Z","shell.execute_reply.started":"2024-02-14T01:55:08.249547Z","shell.execute_reply":"2024-02-14T01:55:10.411654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Addresses, zip codes, and names are high-cardinality variables.","metadata":{}},{"cell_type":"code","source":"display(df_train_person_2.filter(pl.col('num_group1') == 0).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:55:50.824381Z","iopub.execute_input":"2024-02-14T01:55:50.824819Z","iopub.status.idle":"2024-02-14T01:55:51.345441Z","shell.execute_reply.started":"2024-02-14T01:55:50.824784Z","shell.execute_reply":"2024-02-14T01:55:51.344525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_person_2.filter(pl.col('num_group1') != 0).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:56:03.904335Z","iopub.execute_input":"2024-02-14T01:56:03.904922Z","iopub.status.idle":"2024-02-14T01:56:04.281116Z","shell.execute_reply.started":"2024-02-14T01:56:03.904875Z","shell.execute_reply":"2024-02-14T01:56:04.279992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# credit_bureau_a_2","metadata":{}},{"cell_type":"code","source":"df_train_credit_bureau_a_2 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_credit_bureau_a_2_*.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:29:23.203820Z","iopub.execute_input":"2024-02-26T01:29:23.204255Z","iopub.status.idle":"2024-02-26T01:29:23.221690Z","shell.execute_reply.started":"2024-02-26T01:29:23.204225Z","shell.execute_reply":"2024-02-26T01:29:23.220657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_credit_bureau_a_2.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:50:24.902901Z","iopub.execute_input":"2024-02-14T07:50:24.903888Z","iopub.status.idle":"2024-02-14T07:50:24.916337Z","shell.execute_reply.started":"2024-02-14T07:50:24.903850Z","shell.execute_reply":"2024-02-14T07:50:24.914902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_credit_bureau_a_2.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:50:26.501357Z","iopub.execute_input":"2024-02-14T07:50:26.501910Z","iopub.status.idle":"2024-02-14T07:50:26.626444Z","shell.execute_reply.started":"2024-02-14T07:50:26.501877Z","shell.execute_reply":"2024-02-14T07:50:26.625267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_credit_bureau_a_2.select(cs.by_dtype(pl.NUMERIC_DTYPES, pl.Datetime)).collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:50:27.750431Z","iopub.execute_input":"2024-02-14T07:50:27.751406Z","iopub.status.idle":"2024-02-14T07:52:01.326907Z","shell.execute_reply.started":"2024-02-14T07:50:27.751365Z","shell.execute_reply":"2024-02-14T07:52:01.325722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The max of `pmts_dpd_303P` is greater than `600000 days ~ 1643 years`.","metadata":{}},{"cell_type":"code","source":"display(df_train_credit_bureau_a_2.select(cs.by_dtype(pl.String).n_unique()).collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:44:17.424236Z","iopub.execute_input":"2024-02-13T02:44:17.424803Z","iopub.status.idle":"2024-02-13T02:44:51.939514Z","shell.execute_reply.started":"2024-02-13T02:44:17.424756Z","shell.execute_reply":"2024-02-13T02:44:51.937966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_credit_bureau_a_2.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:30:08.293699Z","iopub.execute_input":"2024-02-26T01:30:08.294135Z","iopub.status.idle":"2024-02-26T01:30:19.164040Z","shell.execute_reply.started":"2024-02-26T01:30:08.294094Z","shell.execute_reply":"2024-02-26T01:30:19.162764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_credit_bureau_a_2.group_by('case_id', 'num_group1')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:34:41.895324Z","iopub.execute_input":"2024-02-26T01:34:41.896078Z","iopub.status.idle":"2024-02-26T01:35:36.981203Z","shell.execute_reply.started":"2024-02-26T01:34:41.896037Z","shell.execute_reply":"2024-02-26T01:35:36.980008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_a_2, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:45:02.430293Z","iopub.execute_input":"2024-02-13T02:45:02.430779Z","iopub.status.idle":"2024-02-13T02:45:11.810553Z","shell.execute_reply.started":"2024-02-13T02:45:02.430741Z","shell.execute_reply":"2024-02-13T02:45:11.809180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_a_2, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:45:17.869477Z","iopub.execute_input":"2024-02-13T02:45:17.869926Z","iopub.status.idle":"2024-02-13T02:45:27.450758Z","shell.execute_reply.started":"2024-02-13T02:45:17.869892Z","shell.execute_reply":"2024-02-13T02:45:27.449784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_categorical(df_train_credit_bureau_a_2, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:45:27.452571Z","iopub.execute_input":"2024-02-13T02:45:27.453518Z","iopub.status.idle":"2024-02-13T02:46:34.846779Z","shell.execute_reply.started":"2024-02-13T02:45:27.453483Z","shell.execute_reply":"2024-02-13T02:46:34.845519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_credit_bureau_a_2, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:41:00.878248Z","iopub.execute_input":"2024-02-13T04:41:00.878652Z","iopub.status.idle":"2024-02-13T04:41:06.262453Z","shell.execute_reply.started":"2024-02-13T04:41:00.878624Z","shell.execute_reply":"2024-02-13T04:41:06.261405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr(df_train_credit_bureau_a_2, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T04:41:07.668384Z","iopub.execute_input":"2024-02-13T04:41:07.668774Z","iopub.status.idle":"2024-02-13T04:41:12.251814Z","shell.execute_reply.started":"2024-02-13T04:41:07.668741Z","shell.execute_reply":"2024-02-13T04:41:12.250806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_a_2, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:14:00.822990Z","iopub.execute_input":"2024-02-13T05:14:00.823429Z","iopub.status.idle":"2024-02-13T05:14:06.481753Z","shell.execute_reply.started":"2024-02-13T05:14:00.823396Z","shell.execute_reply":"2024-02-13T05:14:06.480661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_a_2, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:14:07.820439Z","iopub.execute_input":"2024-02-13T05:14:07.820806Z","iopub.status.idle":"2024-02-13T05:14:13.282034Z","shell.execute_reply.started":"2024-02-13T05:14:07.820778Z","shell.execute_reply":"2024-02-13T05:14:13.280965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_a_2, 'M')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:14:14.448728Z","iopub.execute_input":"2024-02-13T05:14:14.449119Z","iopub.status.idle":"2024-02-13T05:15:22.096599Z","shell.execute_reply.started":"2024-02-13T05:14:14.449089Z","shell.execute_reply":"2024-02-13T05:15:22.095449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_a_2.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:22:29.638941Z","iopub.execute_input":"2024-02-14T01:22:29.639388Z","iopub.status.idle":"2024-02-14T01:22:55.786551Z","shell.execute_reply.started":"2024-02-14T01:22:29.639352Z","shell.execute_reply":"2024-02-14T01:22:55.785106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_a_2.group_by('case_id').mean(), 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:23:14.202359Z","iopub.execute_input":"2024-02-14T01:23:14.202828Z","iopub.status.idle":"2024-02-14T01:23:44.105777Z","shell.execute_reply.started":"2024-02-14T01:23:14.202793Z","shell.execute_reply":"2024-02-14T01:23:44.104121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_tbl_rows(-1)\n    for f in glob.iglob('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_credit_bureau_a_2_*.parquet'):\n        display(\n            pl.scan_parquet(f)\n            .join(df_train_base.lazy(), on = 'case_id', how = 'left')\n            .select(pl.col('WEEK_NUM').unique())\n            .collect()\n        )","metadata":{"execution":{"iopub.status.busy":"2024-03-13T01:06:49.134077Z","iopub.execute_input":"2024-03-13T01:06:49.134579Z","iopub.status.idle":"2024-03-13T01:07:03.013838Z","shell.execute_reply.started":"2024-03-13T01:06:49.134542Z","shell.execute_reply":"2024-03-13T01:07:03.012258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# credit_bureau_b_2","metadata":{}},{"cell_type":"code","source":"df_train_credit_bureau_b_2 = scan_data_and_parse_dates('/kaggle/input/home-credit-credit-risk-model-stability/parquet_files/train/train_credit_bureau_b_2.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:31:46.137495Z","iopub.execute_input":"2024-02-26T01:31:46.138015Z","iopub.status.idle":"2024-02-26T01:31:46.156513Z","shell.execute_reply.started":"2024-02-26T01:31:46.137976Z","shell.execute_reply":"2024-02-26T01:31:46.155337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with pl.Config() as cfg:\n    cfg.set_fmt_str_lengths(150)\n    display(get_feature_definitions(df_train_credit_bureau_b_2.columns))","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:48:30.123157Z","iopub.execute_input":"2024-02-14T07:48:30.127663Z","iopub.status.idle":"2024-02-14T07:48:30.147882Z","shell.execute_reply.started":"2024-02-14T07:48:30.127503Z","shell.execute_reply":"2024-02-14T07:48:30.146569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_credit_bureau_b_2.head().collect())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:48:30.690366Z","iopub.execute_input":"2024-02-14T07:48:30.691629Z","iopub.status.idle":"2024-02-14T07:48:30.736248Z","shell.execute_reply.started":"2024-02-14T07:48:30.691579Z","shell.execute_reply":"2024-02-14T07:48:30.735058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train_credit_bureau_b_2.collect().describe())","metadata":{"execution":{"iopub.status.busy":"2024-02-14T07:48:31.345967Z","iopub.execute_input":"2024-02-14T07:48:31.346536Z","iopub.status.idle":"2024-02-14T07:48:33.267519Z","shell.execute_reply.started":"2024-02-14T07:48:31.346471Z","shell.execute_reply":"2024-02-14T07:48:33.266351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The max of `pmts_dpdvalue_108P` is more than `100000000 days ~ 273972 years`.","metadata":{}},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_credit_bureau_b_2.group_by('case_id')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:32:38.201288Z","iopub.execute_input":"2024-02-26T01:32:38.201708Z","iopub.status.idle":"2024-02-26T01:32:39.478035Z","shell.execute_reply.started":"2024-02-26T01:32:38.201676Z","shell.execute_reply":"2024-02-26T01:32:39.476406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    sns.histplot(df_train_credit_bureau_b_2.group_by('case_id', 'num_group1')\n                 .agg(pl.len().alias('count'))\n                 .select('count').collect()['count'].to_pandas(),\n                 bins = 50);","metadata":{"execution":{"iopub.status.busy":"2024-02-26T01:36:20.908041Z","iopub.execute_input":"2024-02-26T01:36:20.908507Z","iopub.status.idle":"2024-02-26T01:36:21.557087Z","shell.execute_reply.started":"2024-02-26T01:36:20.908470Z","shell.execute_reply":"2024-02-26T01:36:21.555551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_b_2, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T02:47:54.357056Z","iopub.execute_input":"2024-02-13T02:47:54.357546Z","iopub.status.idle":"2024-02-13T02:47:59.874608Z","shell.execute_reply.started":"2024-02-13T02:47:54.357513Z","shell.execute_reply":"2024-02-13T02:47:59.873000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical(df_train_credit_bureau_b_2, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T08:01:01.145202Z","iopub.execute_input":"2024-02-14T08:01:01.146141Z","iopub.status.idle":"2024-02-14T08:01:06.552791Z","shell.execute_reply.started":"2024-02-14T08:01:01.146107Z","shell.execute_reply":"2024-02-14T08:01:06.551619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_b_2, 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-13T05:15:22.098695Z","iopub.execute_input":"2024-02-13T05:15:22.099054Z","iopub.status.idle":"2024-02-13T05:15:22.895707Z","shell.execute_reply.started":"2024-02-13T05:15:22.099024Z","shell.execute_reply":"2024-02-13T05:15:22.894679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_null(df_train_credit_bureau_b_2, 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T08:00:57.491182Z","iopub.execute_input":"2024-02-14T08:00:57.492577Z","iopub.status.idle":"2024-02-14T08:00:58.357577Z","shell.execute_reply.started":"2024-02-14T08:00:57.492534Z","shell.execute_reply":"2024-02-14T08:00:58.356379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_b_2.group_by('case_id').mean(), 'A')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:24:27.897812Z","iopub.execute_input":"2024-02-14T01:24:27.898685Z","iopub.status.idle":"2024-02-14T01:24:31.238734Z","shell.execute_reply.started":"2024-02-14T01:24:27.898618Z","shell.execute_reply":"2024-02-14T01:24:31.237435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with warnings.catch_warnings():\n    warnings.simplefilter('ignore')\n    plot_numerical_by_target(df_train_base, df_train_credit_bureau_b_2.group_by('case_id').mean(), 'P')","metadata":{"execution":{"iopub.status.busy":"2024-02-14T01:24:37.219444Z","iopub.execute_input":"2024-02-14T01:24:37.219921Z","iopub.status.idle":"2024-02-14T01:24:40.511533Z","shell.execute_reply.started":"2024-02-14T01:24:37.219885Z","shell.execute_reply":"2024-02-14T01:24:40.510021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# To be continued","metadata":{}},{"cell_type":"markdown","source":"- Visualization. Since the number of features is large, we should select good features to visualize.\n  - E.g. features used in the [Starter Notebook](https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook)\n  - See also <https://www.kaggle.com/code/thomasmeiner/home-credit-eda-feature-engineering-modelling>\n  - We have plotted `A`, `P`, and `M` groups.\n- Bivariate analysis. There are some features with similar/same descriptions. Those can be related.\n  - We have plotted correlations in `A` and `P` groups.\n  - We have plotted distributions of `A` and `P` groups by target. For depth>=1 tables we have used the mean for each `case_id`, but other aggregations might make some sense. Remember that `num_group1` and `num_group2` may have specific meanings.","metadata":{}},{"cell_type":"markdown","source":"# Some consideration","metadata":{}},{"cell_type":"markdown","source":"- Cast string features to `Categorical` to save memory.\n- Use `float32` to further reduce memory.\n- Think about high-cardinality features such as addresses and names.\n- Think about high proportion of null values.\n- Think about target imbalance.\n- Features and target can be time-dependent.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}