{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7602123,"sourceType":"competition"},{"sourceId":7575239,"sourceType":"datasetVersion","datasetId":4405300}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#### This notebook is to focus on missing values patterns.\n\n#### Notebook styling, Overview and Data Schema is taken from [1] (please consider to upvote this notebook as well)\n\n#### References: \n- [1] [home-credit-crms-2024-eda-and-submission](https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission) by [sergiosaharovskiy](sergiosaharovskiy)","metadata":{}},{"cell_type":"code","source":"!wget http://bit.ly/3ZLyF82 -O CSS.css -q\n \nfrom IPython.core.display import HTML\nwith open('./CSS.css', 'r') as file:\n custom_css = file.read()\n\nHTML(custom_css)\n\n!cp /kaggle/input/2024-home-credit-public-repo/HomeCredit2024Banner.png .","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:49:14.448115Z","iopub.execute_input":"2024-02-11T18:49:14.448609Z","iopub.status.idle":"2024-02-11T18:49:16.981438Z","shell.execute_reply.started":"2024-02-11T18:49:14.448566Z","shell.execute_reply":"2024-02-11T18:49:16.979986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Libraries

","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/2024-home-credit-public-repo')\nimport os, gc \nimport subprocess\nimport numpy as np\nimport pandas as pd\nimport polars as pl\n\nfrom glob import glob\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport missingno as msno\nfrom pathlib import Path\nfrom tqdm.auto import tqdm\nfrom utils import RC, PALETTE, cS\nfrom utils import plot_count\n\nsns.set(rc=RC)\n\nimport warnings\nwarnings.filterwarnings('ignore')\npd.set_option('display.max_colwidth', None)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:08:51.868446Z","iopub.execute_input":"2024-02-11T19:08:51.870957Z","iopub.status.idle":"2024-02-11T19:08:51.885353Z","shell.execute_reply.started":"2024-02-11T19:08:51.870886Z","shell.execute_reply":"2024-02-11T19:08:51.883667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Overview

\n\nThe goal of this competition is to predict which clients are more likely to default on their loans. The evaluation will favor solutions that are stable over time.\n\nYour participation may offer consumer finance providers a more reliable and longer-lasting way to assess a potential client’s default risk.","metadata":{}},{"cell_type":"markdown","source":"##

Data Schema

","metadata":{}},{"cell_type":"code","source":"!tree '/kaggle/input/home-credit-credit-risk-model-stability' -d \n\nROOT = '/kaggle/input/home-credit-credit-risk-model-stability'\nfolders = ['csv_files/train', 'parquet_files/train', 'csv_files/test', 'parquet_files/test']\nextensions = ['.csv', '.parquet'] * 2\n\nfor dir_, ext in zip(folders, extensions):\n folder_path = Path(ROOT) / dir_\n num_files = len(list(folder_path.glob(f'*{ext}')))\n print(f\"{cS.blk}Number of .csv files in: {'./' + dir_:>21}: {cS.blu}{num_files}{cS.res}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:49:20.359256Z","iopub.execute_input":"2024-02-11T18:49:20.360036Z","iopub.status.idle":"2024-02-11T18:49:21.435711Z","shell.execute_reply.started":"2024-02-11T18:49:20.360003Z","shell.execute_reply":"2024-02-11T18:49:21.434194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n \n

","metadata":{"execution":{"iopub.status.busy":"2024-02-06T15:39:34.368459Z","iopub.execute_input":"2024-02-06T15:39:34.369137Z","iopub.status.idle":"2024-02-06T15:39:34.406577Z","shell.execute_reply.started":"2024-02-06T15:39:34.36909Z","shell.execute_reply":"2024-02-06T15:39:34.405212Z"}}},{"cell_type":"markdown","source":"- There are **465** features and **436** respective descriptions in `feature_definitions.csv`. There are no missing descriptions so it means some feature might have same descriptions (for example description `Number of tax deductions` for features: `pmtcount_4527229L`, `pmtcount_4955617L`, `pmtcount_693L`).\n\n- Various predictors were transformed, so to have the following notation for similar groups of transformations: **`P M A D T L`**\n\n- Above you can see that `.csv` files duplicate `.parquet` ones. Ideally, we gotta check if they are really the same as mentioned in the competition data sections.","metadata":{}},{"cell_type":"markdown","source":"##

Tables

","metadata":{}},{"cell_type":"code","source":"def get_disk_usage(directory):\n cmd = f'du {directory}/* -h | sort -rh'\n result = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, text=True)\n output_lines = result.stdout.split('\\n')\n\n # Extract file/directory names and sizes\n data = [line.split('\\t') for line in output_lines if line]\n df = pd.DataFrame(data, columns=['size', 'path'])\n df['file_name'] = df.path.str.replace('train_|test_', '', regex=True).\\\n apply(lambda x: Path(x).stem)\n return df\n\n# train_disk_usage = get_disk_usage(f'{ROOT}/csv_files/train').reset_index()\n# test_disk_usage = get_disk_usage(f'{ROOT}/csv_files/test')\n\n# train_disk_usage.reset_index().merge(test_disk_usage, on=['file_name'],\n# how='outer', suffixes=['_train', '_test'])\\\n# .sort_values(by='index').drop(columns=['index'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:49:21.439984Z","iopub.execute_input":"2024-02-11T18:49:21.440909Z","iopub.status.idle":"2024-02-11T18:49:21.449400Z","shell.execute_reply.started":"2024-02-11T18:49:21.440854Z","shell.execute_reply":"2024-02-11T18:49:21.448345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_file(path, depth=None, vb=False):\n df = pl.read_parquet(path)\n df = df.pipe(Pipeline.set_table_dtypes)\n \n if depth in [1, 2]:\n df = df.group_by(\"case_id\").agg(Aggregator.get_exprs(df))\n \n if vb:\n print('load from:', path)\n print(df.head(2))\n \n return df\n\ndef read_files(regex_path, depth=None, vb=False):\n chunks = []\n for path in glob(str(regex_path)):\n chunks.append(pl.read_parquet(path).pipe(Pipeline.set_table_dtypes))\n \n df = pl.concat(chunks, how=\"vertical_relaxed\")\n if depth in [1, 2]:\n df = df.group_by(\"case_id\").agg(Aggregator.get_exprs(df))\n \n if vb:\n print('load from:', path)\n print(df.head(2))\n return df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:49:21.451615Z","iopub.execute_input":"2024-02-11T18:49:21.452676Z","iopub.status.idle":"2024-02-11T18:49:21.466592Z","shell.execute_reply.started":"2024-02-11T18:49:21.452633Z","shell.execute_reply":"2024-02-11T18:49:21.465518Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Pipeline:\n @staticmethod\n def set_table_dtypes(df):\n for col in df.columns:\n if col in [\"case_id\", \"WEEK_NUM\", \"num_group1\", \"num_group2\"]:\n df = df.with_columns(pl.col(col).cast(pl.Int64))\n elif col in [\"date_decision\"]:\n df = df.with_columns(pl.col(col).cast(pl.Date))\n elif col[-1] in (\"P\", \"A\"):\n df = df.with_columns(pl.col(col).cast(pl.Float64))\n elif col[-1] in (\"M\",):\n df = df.with_columns(pl.col(col).cast(pl.String))\n elif col[-1] in (\"D\",):\n df = df.with_columns(pl.col(col).cast(pl.Date)) \n\n return df\n \n @staticmethod\n def handle_dates(df):\n for col in df.columns:\n if col[-1] in (\"D\",):\n df = df.with_columns(pl.col(col) - pl.col(\"date_decision\"))\n df = df.with_columns(pl.col(col).dt.total_days())\n \n df = df.drop(\"date_decision\", \"MONTH\")\n\n return df\n \n @staticmethod\n def filter_cols(df):\n for col in df.columns:\n if col not in [\"target\", \"case_id\", \"WEEK_NUM\"]:\n isnull = df[col].is_null().mean()\n\n if isnull > 0.95:\n df = df.drop(col)\n\n for col in df.columns:\n if (col not in [\"target\", \"case_id\", \"WEEK_NUM\"]) & (df[col].dtype == pl.String):\n freq = df[col].n_unique()\n\n if (freq == 1) | (freq > 200):\n df = df.drop(col)\n\n return df","metadata":{"execution":{"iopub.status.busy":"2024-02-11T18:50:05.490800Z","iopub.execute_input":"2024-02-11T18:50:05.491354Z","iopub.status.idle":"2024-02-11T18:50:05.504893Z","shell.execute_reply.started":"2024-02-11T18:50:05.491311Z","shell.execute_reply":"2024-02-11T18:50:05.503486Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ### Automatic Aggregation\nclass Aggregator:\n\n num_aggregators = [pl.max, pl.min, pl.first, pl.last, pl.mean]\n str_aggregators = [pl.max, pl.min, pl.first, pl.last, pl.n_unique]\n \n @staticmethod\n def num_expr(df):\n cols = [col for col in df.columns if col[-1] in (\"P\", \"A\")]\n expr_all = []\n for method in Aggregator.num_aggregators:\n expr = [method(col).alias(f\"{method.__name__}_{col}\") for col in cols]\n expr_all += expr\n\n return expr_all\n\n @staticmethod\n def date_expr(df):\n cols = [col for col in df.columns if col[-1] in (\"D\",)]\n expr_all = []\n for method in Aggregator.num_aggregators:\n expr = [method(col).alias(f\"{method.__name__}_{col}\") for col in cols] \n expr_all += expr\n\n return expr_all\n\n @staticmethod\n def str_expr(df):\n cols = [col for col in df.columns if col[-1] in (\"M\",)]\n \n expr_all = []\n for method in Aggregator.str_aggregators:\n expr = [method(col).alias(f\"{method.__name__}_{col}\") for col in cols] \n expr_all += expr\n \n expr_mode = [\n pl.col(col)\n .drop_nulls()\n .mode()\n .first()\n .alias(f\"mode_{col}\")\n for col in cols\n ]\n\n return expr_all + expr_mode\n\n @staticmethod\n def other_expr(df):\n cols = [col for col in df.columns if col[-1] in (\"T\", \"L\")]\n \n expr_all = []\n for method in Aggregator.str_aggregators:\n expr = [method(col).alias(f\"{method.__name__}_{col}\") for col in cols] \n expr_all += expr\n\n return expr_all\n \n @staticmethod\n def count_expr(df):\n cols = [col for col in df.columns if \"num_group\" in col]\n\n expr_all = []\n for method in Aggregator.str_aggregators:\n expr = [method(col).alias(f\"{method.__name__}_{col}\") for col in cols] \n expr_all += expr\n\n return expr_all\n\n @staticmethod\n def get_exprs(df):\n exprs = Aggregator.num_expr(df)\\\n + Aggregator.date_expr(df)\\\n + Aggregator.str_expr(df)\\\n + Aggregator.other_expr(df)\\\n + Aggregator.count_expr(df)\n return exprs","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:55:07.713457Z","iopub.execute_input":"2024-02-11T18:55:07.713954Z","iopub.status.idle":"2024-02-11T18:55:07.888915Z","shell.execute_reply.started":"2024-02-11T18:55:07.713913Z","shell.execute_reply":"2024-02-11T18:55:07.887635Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## base file","metadata":{}},{"cell_type":"code","source":"ROOT = Path(\"/kaggle/input/home-credit-credit-risk-model-stability\")\nTRAIN_DIR = ROOT / \"parquet_files\" / \"train\"\nPATH_BASE_TRAIN = f'{ROOT}/csv_files/train/train_base.csv'\nPATH_BASE_TEST = f'{ROOT}/csv_files/test/test_base.csv'\n\ntrain = pd.read_csv(PATH_BASE_TRAIN)\n# test = pd.read_csv(PATH_BASE_TEST)\n\ndisplay(train.head(3))\ndisplay(train.dtypes)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T18:49:21.468012Z","iopub.execute_input":"2024-02-11T18:49:21.468429Z","iopub.status.idle":"2024-02-11T18:49:22.735797Z","shell.execute_reply.started":"2024-02-11T18:49:21.468388Z","shell.execute_reply":"2024-02-11T18:49:22.734440Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Static (d=0)

","metadata":{}},{"cell_type":"code","source":"df_static = pl.concat([\n read_file(TRAIN_DIR / \"train_static_0_0.parquet\", 0, True),\n read_file(TRAIN_DIR / \"train_static_0_1.parquet\", 0, True),\n], how=\"vertical_relaxed\")\n\nprint('df_static shape:', df_static.shape)\nprint('unique case_id:', df_static.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:53:04.693342Z","iopub.execute_input":"2024-02-11T18:53:04.694522Z","iopub.status.idle":"2024-02-11T18:53:11.069728Z","shell.execute_reply.started":"2024-02-11T18:53:04.694469Z","shell.execute_reply":"2024-02-11T18:53:11.068776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_static.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:53:11.071398Z","iopub.execute_input":"2024-02-11T18:53:11.071895Z","iopub.status.idle":"2024-02-11T18:54:10.172290Z","shell.execute_reply.started":"2024-02-11T18:53:11.071867Z","shell.execute_reply":"2024-02-11T18:54:10.171282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_static.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:54:10.173696Z","iopub.execute_input":"2024-02-11T18:54:10.174247Z","iopub.status.idle":"2024-02-11T18:54:18.286558Z","shell.execute_reply.started":"2024-02-11T18:54:10.174216Z","shell.execute_reply":"2024-02-11T18:54:18.284490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Static_cb (d=0)

","metadata":{}},{"cell_type":"code","source":"df_static_cb = read_file(TRAIN_DIR / \"train_static_cb_0.parquet\", 0, True)\n\nprint('df_static_cb shape:', df_static_cb.shape)\nprint('unique case_id:', df_static_cb.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:52:18.750805Z","iopub.execute_input":"2024-02-11T18:52:18.751285Z","iopub.status.idle":"2024-02-11T18:52:20.377823Z","shell.execute_reply.started":"2024-02-11T18:52:18.751244Z","shell.execute_reply":"2024-02-11T18:52:20.376580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_static_cb.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:52:24.611613Z","iopub.execute_input":"2024-02-11T18:52:24.612100Z","iopub.status.idle":"2024-02-11T18:52:43.661105Z","shell.execute_reply.started":"2024-02-11T18:52:24.612052Z","shell.execute_reply":"2024-02-11T18:52:43.659364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_static_cb.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:51:57.164146Z","iopub.execute_input":"2024-02-11T18:51:57.165028Z","iopub.status.idle":"2024-02-11T18:51:59.923126Z","shell.execute_reply.started":"2024-02-11T18:51:57.164974Z","shell.execute_reply":"2024-02-11T18:51:59.921902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_static_cb\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:54:37.392610Z","iopub.execute_input":"2024-02-11T18:54:37.394009Z","iopub.status.idle":"2024-02-11T18:54:37.628768Z","shell.execute_reply.started":"2024-02-11T18:54:37.393952Z","shell.execute_reply":"2024-02-11T18:54:37.627615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Person_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_person_1 = read_file(TRAIN_DIR / \"train_person_1.parquet\", 1, True)\n\nprint('df_person_1 shape:', df_person_1.shape)\nprint('unique case_id:', df_person_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:55:14.056481Z","iopub.execute_input":"2024-02-11T18:55:14.056979Z","iopub.status.idle":"2024-02-11T18:56:21.819337Z","shell.execute_reply.started":"2024-02-11T18:55:14.056944Z","shell.execute_reply":"2024-02-11T18:56:21.817902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_person_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:56:21.821863Z","iopub.execute_input":"2024-02-11T18:56:21.822346Z","iopub.status.idle":"2024-02-11T18:57:53.906917Z","shell.execute_reply.started":"2024-02-11T18:56:21.822301Z","shell.execute_reply":"2024-02-11T18:57:53.903585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_person_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:57:53.910284Z","iopub.execute_input":"2024-02-11T18:57:53.915528Z","iopub.status.idle":"2024-02-11T18:58:12.140396Z","shell.execute_reply.started":"2024-02-11T18:57:53.915469Z","shell.execute_reply":"2024-02-11T18:58:12.138608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_person_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:12.143532Z","iopub.execute_input":"2024-02-11T18:58:12.143996Z","iopub.status.idle":"2024-02-11T18:58:12.624409Z","shell.execute_reply.started":"2024-02-11T18:58:12.143955Z","shell.execute_reply":"2024-02-11T18:58:12.622906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Deposit_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_deposit_1 = read_file(TRAIN_DIR / \"train_deposit_1.parquet\", depth=1, vb=True)\n\nprint('df_deposit_1 shape:', df_deposit_1.shape)\nprint('unique case_id:', df_deposit_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:12.626456Z","iopub.execute_input":"2024-02-11T18:58:12.627347Z","iopub.status.idle":"2024-02-11T18:58:12.846531Z","shell.execute_reply.started":"2024-02-11T18:58:12.627298Z","shell.execute_reply":"2024-02-11T18:58:12.845520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_deposit_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:12.847740Z","iopub.execute_input":"2024-02-11T18:58:12.848771Z","iopub.status.idle":"2024-02-11T18:58:14.039040Z","shell.execute_reply.started":"2024-02-11T18:58:12.848715Z","shell.execute_reply":"2024-02-11T18:58:14.038083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_deposit_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:14.040335Z","iopub.execute_input":"2024-02-11T18:58:14.041269Z","iopub.status.idle":"2024-02-11T18:58:15.538738Z","shell.execute_reply.started":"2024-02-11T18:58:14.041236Z","shell.execute_reply":"2024-02-11T18:58:15.536933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_deposit_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:15.540575Z","iopub.execute_input":"2024-02-11T18:58:15.541008Z","iopub.status.idle":"2024-02-11T18:58:15.742686Z","shell.execute_reply.started":"2024-02-11T18:58:15.540970Z","shell.execute_reply":"2024-02-11T18:58:15.741331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Debit_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_debit_1 = read_file(TRAIN_DIR / \"train_debitcard_1.parquet\", depth=1, vb=True)\n\nprint('df_debit_1 shape:', df_debit_1.shape)\nprint('unique case_id:', df_debit_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:15.744173Z","iopub.execute_input":"2024-02-11T18:58:15.744561Z","iopub.status.idle":"2024-02-11T18:58:15.934346Z","shell.execute_reply.started":"2024-02-11T18:58:15.744530Z","shell.execute_reply":"2024-02-11T18:58:15.933282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_debit_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:15.938019Z","iopub.execute_input":"2024-02-11T18:58:15.939157Z","iopub.status.idle":"2024-02-11T18:58:17.358743Z","shell.execute_reply.started":"2024-02-11T18:58:15.939112Z","shell.execute_reply":"2024-02-11T18:58:17.357456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_debit_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:17.360209Z","iopub.execute_input":"2024-02-11T18:58:17.360592Z","iopub.status.idle":"2024-02-11T18:58:19.092573Z","shell.execute_reply.started":"2024-02-11T18:58:17.360558Z","shell.execute_reply":"2024-02-11T18:58:19.091617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_debit_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:19.094235Z","iopub.execute_input":"2024-02-11T18:58:19.094825Z","iopub.status.idle":"2024-02-11T18:58:19.279060Z","shell.execute_reply.started":"2024-02-11T18:58:19.094789Z","shell.execute_reply":"2024-02-11T18:58:19.277907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Credit_Bureau_b_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_credit_bureau_b_1 = read_file(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\", depth=1, vb=True)\n\nprint('df_credit_bureau_b_1 shape:', df_credit_bureau_b_1.shape)\nprint('unique case_id:', df_credit_bureau_b_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:19.280775Z","iopub.execute_input":"2024-02-11T18:58:19.281646Z","iopub.status.idle":"2024-02-11T18:58:20.583550Z","shell.execute_reply.started":"2024-02-11T18:58:19.281606Z","shell.execute_reply":"2024-02-11T18:58:20.582474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_credit_bureau_b_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:20.584688Z","iopub.execute_input":"2024-02-11T18:58:20.585692Z","iopub.status.idle":"2024-02-11T18:58:23.190122Z","shell.execute_reply.started":"2024-02-11T18:58:20.585660Z","shell.execute_reply":"2024-02-11T18:58:23.188715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_credit_bureau_b_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:23.191929Z","iopub.execute_input":"2024-02-11T18:58:23.192378Z","iopub.status.idle":"2024-02-11T18:58:29.475973Z","shell.execute_reply.started":"2024-02-11T18:58:23.192343Z","shell.execute_reply":"2024-02-11T18:58:29.474309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_credit_bureau_b_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:29.478097Z","iopub.execute_input":"2024-02-11T18:58:29.478702Z","iopub.status.idle":"2024-02-11T18:58:29.782543Z","shell.execute_reply.started":"2024-02-11T18:58:29.478664Z","shell.execute_reply":"2024-02-11T18:58:29.781178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Tax_registry_a_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_tax_registry_a_1 = read_file(TRAIN_DIR / \"train_tax_registry_a_1.parquet\", depth=1, vb=True)\n\nprint('df_tax_registry_a_1 shape:', df_tax_registry_a_1.shape)\nprint('unique case_id:', df_tax_registry_a_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:29.784176Z","iopub.execute_input":"2024-02-11T18:58:29.784687Z","iopub.status.idle":"2024-02-11T18:58:33.690546Z","shell.execute_reply.started":"2024-02-11T18:58:29.784647Z","shell.execute_reply":"2024-02-11T18:58:33.689053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_tax_registry_a_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:33.692032Z","iopub.execute_input":"2024-02-11T18:58:33.692734Z","iopub.status.idle":"2024-02-11T18:58:37.597234Z","shell.execute_reply.started":"2024-02-11T18:58:33.692672Z","shell.execute_reply":"2024-02-11T18:58:37.595929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_tax_registry_a_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:37.599223Z","iopub.execute_input":"2024-02-11T18:58:37.599620Z","iopub.status.idle":"2024-02-11T18:58:39.798931Z","shell.execute_reply.started":"2024-02-11T18:58:37.599584Z","shell.execute_reply":"2024-02-11T18:58:39.797546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_tax_registry_a_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:39.801116Z","iopub.execute_input":"2024-02-11T18:58:39.801887Z","iopub.status.idle":"2024-02-11T18:58:39.994773Z","shell.execute_reply.started":"2024-02-11T18:58:39.801845Z","shell.execute_reply":"2024-02-11T18:58:39.993488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Tax_registry_b_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_tax_registry_b_1 = read_file(TRAIN_DIR / \"train_tax_registry_b_1.parquet\", depth=1, vb=True)\n\nprint('df_tax_registry_b_1 shape:', df_tax_registry_b_1.shape)\nprint('unique case_id:', df_tax_registry_b_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:39.996679Z","iopub.execute_input":"2024-02-11T18:58:39.997094Z","iopub.status.idle":"2024-02-11T18:58:41.383865Z","shell.execute_reply.started":"2024-02-11T18:58:39.997039Z","shell.execute_reply":"2024-02-11T18:58:41.382614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_tax_registry_b_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:41.385456Z","iopub.execute_input":"2024-02-11T18:58:41.385830Z","iopub.status.idle":"2024-02-11T18:58:42.963760Z","shell.execute_reply.started":"2024-02-11T18:58:41.385800Z","shell.execute_reply":"2024-02-11T18:58:42.962333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_tax_registry_b_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:42.965481Z","iopub.execute_input":"2024-02-11T18:58:42.965880Z","iopub.status.idle":"2024-02-11T18:58:44.639944Z","shell.execute_reply.started":"2024-02-11T18:58:42.965846Z","shell.execute_reply":"2024-02-11T18:58:44.637153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_tax_registry_b_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:44.641727Z","iopub.execute_input":"2024-02-11T18:58:44.642116Z","iopub.status.idle":"2024-02-11T18:58:44.821958Z","shell.execute_reply.started":"2024-02-11T18:58:44.642081Z","shell.execute_reply":"2024-02-11T18:58:44.821037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Tax_registry_c_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_tax_registry_c_1 = read_file(TRAIN_DIR / \"train_tax_registry_c_1.parquet\", depth=1, vb=True)\n\nprint('df_tax_registry_c_1 shape:', df_tax_registry_c_1.shape)\nprint('unique case_id:', df_tax_registry_c_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:44.823338Z","iopub.execute_input":"2024-02-11T18:58:44.823861Z","iopub.status.idle":"2024-02-11T18:58:49.259915Z","shell.execute_reply.started":"2024-02-11T18:58:44.823828Z","shell.execute_reply":"2024-02-11T18:58:49.258543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_tax_registry_c_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:49.261395Z","iopub.execute_input":"2024-02-11T18:58:49.261749Z","iopub.status.idle":"2024-02-11T18:58:53.263163Z","shell.execute_reply.started":"2024-02-11T18:58:49.261719Z","shell.execute_reply":"2024-02-11T18:58:53.261931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_tax_registry_c_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:53.265061Z","iopub.execute_input":"2024-02-11T18:58:53.265843Z","iopub.status.idle":"2024-02-11T18:58:55.550653Z","shell.execute_reply.started":"2024-02-11T18:58:53.265799Z","shell.execute_reply":"2024-02-11T18:58:55.549471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_tax_registry_c_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T18:58:55.556224Z","iopub.execute_input":"2024-02-11T18:58:55.556698Z","iopub.status.idle":"2024-02-11T18:58:55.745286Z","shell.execute_reply.started":"2024-02-11T18:58:55.556660Z","shell.execute_reply":"2024-02-11T18:58:55.743962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Applprev_1_* (d=1)

","metadata":{}},{"cell_type":"code","source":"df_applprev_1 = read_files(TRAIN_DIR / \"train_applprev_1_*.parquet\", depth=1, vb=True)\n\nprint('df_applprev_1 shape:', df_applprev_1.shape)\nprint('unique case_id:', df_applprev_1.to_pandas().case_id.nunique())","metadata":{"execution":{"iopub.status.busy":"2024-02-11T19:09:08.560012Z","iopub.execute_input":"2024-02-11T19:09:08.561457Z","iopub.status.idle":"2024-02-11T19:10:08.747373Z","shell.execute_reply.started":"2024-02-11T19:09:08.561405Z","shell.execute_reply":"2024-02-11T19:10:08.746134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_applprev_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:10:08.750004Z","iopub.execute_input":"2024-02-11T19:10:08.750824Z","iopub.status.idle":"2024-02-11T19:11:09.010866Z","shell.execute_reply.started":"2024-02-11T19:10:08.750779Z","shell.execute_reply":"2024-02-11T19:11:09.009501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_applprev_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:09.012608Z","iopub.execute_input":"2024-02-11T19:11:09.013031Z","iopub.status.idle":"2024-02-11T19:11:21.044810Z","shell.execute_reply.started":"2024-02-11T19:11:09.012993Z","shell.execute_reply":"2024-02-11T19:11:21.042788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_applprev_1\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:21.071532Z","iopub.execute_input":"2024-02-11T19:11:21.072218Z","iopub.status.idle":"2024-02-11T19:11:21.568465Z","shell.execute_reply.started":"2024-02-11T19:11:21.072178Z","shell.execute_reply":"2024-02-11T19:11:21.567381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Other_1 (d=1)

","metadata":{}},{"cell_type":"code","source":"df_other_1 = read_file(TRAIN_DIR / \"train_other_1.parquet\", depth=1, vb=True)\n\nprint('df_other_1 shape:', df_other_1.shape)\nprint('unique case_id:', df_other_1.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:21.570503Z","iopub.execute_input":"2024-02-11T19:11:21.571334Z","iopub.status.idle":"2024-02-11T19:11:21.669780Z","shell.execute_reply.started":"2024-02-11T19:11:21.571285Z","shell.execute_reply":"2024-02-11T19:11:21.668686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_other_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:21.671495Z","iopub.execute_input":"2024-02-11T19:11:21.672028Z","iopub.status.idle":"2024-02-11T19:11:22.815654Z","shell.execute_reply.started":"2024-02-11T19:11:21.671977Z","shell.execute_reply":"2024-02-11T19:11:22.814294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_other_1.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:22.817265Z","iopub.execute_input":"2024-02-11T19:11:22.817638Z","iopub.status.idle":"2024-02-11T19:11:24.799369Z","shell.execute_reply.started":"2024-02-11T19:11:22.817604Z","shell.execute_reply":"2024-02-11T19:11:24.797942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##

Credit_Bureau_b_2 (d=2)

","metadata":{}},{"cell_type":"code","source":"df_credit_bureau_b_2 = read_file(TRAIN_DIR / \"train_credit_bureau_b_2.parquet\", depth=2, vb=True)\n\nprint('df_credit_bureau_b_2 shape:', df_credit_bureau_b_2.shape)\nprint('unique case_id:', df_credit_bureau_b_2.to_pandas().case_id.nunique())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:24.801085Z","iopub.execute_input":"2024-02-11T19:11:24.801590Z","iopub.status.idle":"2024-02-11T19:11:25.295018Z","shell.execute_reply.started":"2024-02-11T19:11:24.801544Z","shell.execute_reply":"2024-02-11T19:11:25.293998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(df_credit_bureau_b_2.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:25.297969Z","iopub.execute_input":"2024-02-11T19:11:25.299518Z","iopub.status.idle":"2024-02-11T19:11:26.270568Z","shell.execute_reply.started":"2024-02-11T19:11:25.299468Z","shell.execute_reply":"2024-02-11T19:11:26.269612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.bar(df_credit_bureau_b_2.to_pandas())\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:26.271860Z","iopub.execute_input":"2024-02-11T19:11:26.272937Z","iopub.status.idle":"2024-02-11T19:11:27.963610Z","shell.execute_reply.started":"2024-02-11T19:11:26.272889Z","shell.execute_reply":"2024-02-11T19:11:27.962385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_credit_bureau_b_2\ngc.collect()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T19:11:27.965152Z","iopub.execute_input":"2024-02-11T19:11:27.965553Z","iopub.status.idle":"2024-02-11T19:11:28.208572Z","shell.execute_reply.started":"2024-02-11T19:11:27.965519Z","shell.execute_reply":"2024-02-11T19:11:28.207119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## from original nb: www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission\n# # Code snippet for null values calculation can be found at \n# # https://www.kaggle.com/datasets/sergiosaharovskiy/2024-home-credit-public-repo\n# # I don't want to waste 1 min 30 second of your life staring at the tqdm bar.\n# train_disk_usage = pd.read_csv('/kaggle/input/2024-home-credit-public-repo/files/train_disk_usage.csv')\n# train_disk_usage.head(3)\n\n# values = train_disk_usage['isna_%'].values.tolist()\n# total_area = train_disk_usage['height'] * train_disk_usage['width']\n# total_area_scaled = total_area / total_area.max()\n\n# rows = 4\n# cols = 8\n# fig, axs = plt.subplots(rows, cols, figsize=(18, 11))\n\n# for i, ax in enumerate(axs.flat):\n \n# outer_square_side = np.sqrt(total_area_scaled[i])\n# inner_square_side = np.sqrt(total_area_scaled[i]*values[i])\n\n# # Add the small square inside the 1x1 image\n# ax.add_patch(plt.Rectangle((0.5 - outer_square_side / 2, 0.5 - outer_square_side / 2),\n# outer_square_side, outer_square_side,\n# color='#F03F47', label='Total Records'))\n \n# ax.add_patch(plt.Rectangle((0.4 - inner_square_side / 2, 0.6 - inner_square_side / 2),\n# inner_square_side, inner_square_side,\n# color='#645F64', label='Null Values'))\n\n# ax.set_xticks([])\n# ax.set_yticks([])\n# ax.set_aspect('equal')\n# ax.set_title(f'{train_disk_usage.file_name.iloc[i]}\\n'\n# f'{train_disk_usage[\"size\"].iloc[i]:}\\nNull_%: {values[i]*100:.2f}')\n\n# plt.legend(bbox_to_anchor=(-4, -.4), loc='lower center', ncol=2)\n# plt.suptitle('\\nNull values% in Train files scaled and shaped as Squares')\n\n# plt.tight_layout()\n# plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T18:49:25.477248Z","iopub.status.idle":"2024-02-11T18:49:25.477592Z","shell.execute_reply.started":"2024-02-11T18:49:25.477425Z","shell.execute_reply":"2024-02-11T18:49:25.477440Z"},"trusted":true},"execution_count":null,"outputs":[]}]}