{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30664,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<center><div style=\"font-size:30px\">Home Credit - Credit Risk Model Stability</div></center>\n\n- 赛题名称：Home Credit - Credit Risk Model Stability\n- 赛题任务：预测客户是否可能拖欠贷款\n- 赛题类型：数据挖掘\n- 赛题链接：https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability","metadata":{}},{"cell_type":"markdown","source":"# 赛题背景\n\n赛题背景\n1997年成立的Home Credit是一家国际性的消费金融服务提供商，专注于为几乎没有信用记录的人提供负责任的贷款服务。传统的信用评估方法对于那些几乎没有信用记录的人可能不够适用。缺乏信用记录可能意味着年龄较小或更倾向于使用现金支付，这使得他们在传统数据方面缺乏足够的信息。缺乏传统数据的情况下，很可能会导致这些人被拒绝贷款申请。因此，数据科学有望帮助更好地预测借款人的偿还能力，从而使那些最需要贷款支持的人群能够更容易地获得贷款。\n\n现实世界中客户的行为不断变化，因此每个评分卡都必须定期更新，这需要时间。评分卡在未来的稳定性至关重要，因为性能突然下降意味着贷款平均会发放给更差的客户。然而，贷款提供商无法在贷款到期日可观察到之前就发现潜在的问题。鉴于重新开发、验证和实施评分卡所需的时间，稳定性是非常可取的。模型稳定性与性能之间存在一种权衡关系，在部署之前必须达到平衡。\n\nHome Credit通过与Kaggle等合作伙伴的比赛，旨在吸引数据科学家和机器学习专家的参与，以开发更准确、更稳定的评分卡模型，从而提高贷款申请的接受率，并改善那些因缺乏信用记录而被拒绝贷款的人群的生活。","metadata":{}},{"cell_type":"markdown","source":"# 赛题任务\n比赛的任务是预测哪些客户更有可能违约贷款。评估将偏向那些随着时间稳定的解决方案。参与者的参与可能会为消费金融服务提供商提供一种更可靠、更持久的方式来评估潜在客户的违约风险。","metadata":{}},{"cell_type":"markdown","source":"# 评价指标（旧）\n\nSubmissions are evaluated using a gini stability metric. A gini score is calculated for predictions corresponding to each WEEK_NUM .\n$$\n\\text { gini }=2 * \\mathrm{AUC}-1\n$$\n\nA linear regression, $a \\cdot x+b$, is fit through the weekly gini scores, and a falling_rate is calculated as $\\min (0, a)$. This is used to penalize models that drop off in predictive ability.\n\nFinally, the variability of the predictions are calculated by taking the standard deviation of the residuals from the above linear regression, applying a penalty to model variablity.\n\nThe final metric is calculated as\n$$\n\\text { stability metric }=\\operatorname{mean}(\\operatorname{gini})+88.0 \\cdot \\min (0, a)-0.5 \\cdot \\operatorname{std}(\\text { residuals })\n$$\n\n\n1. **Gini稳定性指标（Gini Stability Metric）**：\n   - Gini指数是一个衡量预测模型公平性或多样性的统计量，其值介于0和1之间。在预测模型的上下文中，Gini指数可以用来衡量模型预测的准确性。Gini指数越低，表示模型的预测性能越好。\n   - 在这个评价体系中，对于每个星期（WEEK_NUM）的预测，都会计算一个Gini分数。这个分数反映了模型在该周的预测性能。\n\n2. **Gini分数的计算**：\n   - Gini分数的计算公式是 \\( \\text{gini} = 2 \\cdot \\text{AUC} - 1 \\)，其中AUC（Area Under the Curve）是接收者操作特征曲线（ROC Curve）下的面积，它是一个衡量二分类模型性能的指标。AUC的值通常在0.5（随机猜测）到1（完美预测）之间。Gini分数的计算实际上是将AUC映射到0到2的范围内，以便更容易地与其他部分结合。\n\n3. **线性回归和下降率（Falling Rate）**：\n   - 使用线性回归模型拟合每周的Gini分数，可以得到一个斜率（a）和一个截距（b）。斜率代表了Gini分数随时间的变化趋势。如果斜率为正，表示模型的预测性能随时间提高；如果为负，则表示性能下降。\n   - 下降率（falling_rate）是斜率的最小值（min(0, a)），它用于惩罚那些随时间预测性能下降的模型。如果模型的预测性能没有下降，下降率就是0。\n\n4. **预测变异性（Model Variability）**：\n   - 通过计算线性回归残差的标准差（std(residuals)）来衡量模型预测的变异性。这个标准差反映了模型预测值与实际值之间的波动程度。为了惩罚模型的高变异性，通常会对标准差进行一定的处理。\n\n5. **最终评价指标（Stability Metric）**：\n   - 最终的评价指标是稳定性指标，它结合了模型的平均Gini分数、下降率和预测变异性。\n   - 计算公式为：\\( \\text{stability metric} = \\text{mean(gini)} + 88.0 \\cdot \\min(0, a) - 0.5 \\cdot \\text{std(residuals)} \\)。\n   - 这个指标的目的是找到一个在时间上稳定、预测性能高且变异性低的模型。平均Gini分数反映了整体预测能力，下降率惩罚了性能随时间下降的模型，而标准差的惩罚则针对那些预测结果波动较大的模型。\n\n这个评价指标的设计旨在鼓励模型在长期内保持稳定的预测性能，同时尽量减少预测误差。通过这样的评价体系，可以更好地比较和选择不同的预测模型。\n\n\n<div class=\"alert alert-block alert-info\">\n指标存在的问题\n</div>\n\n> https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476449\n\n> https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/482474\n\n```python\ncondition = df_subm['WEEK_NUM'] < (df_subm['WEEK_NUM'].max()-df_subm['WEEK_NUM'].min())/2+df_subm['WEEK_NUM'].min() df_subm.loc[condition, 'score'] = (df_subm.loc[condition, 'score'] - 0.02).clip(0) df_subm=df_subm[[\"case_id\",\"score\"]] df_subm = df_subm.set_index(\"case_id\") df_subm.to_csv(\"submission.csv\")\n```\n\n举个例子，考虑一个基于 Gini 稳定性度量的评价指标，其中的一个关键项是 $88 \\times \\min(0, a)$。这个项的存在使得参赛者有可能采取一些不正当手段来提高他们的得分。**例如，他们可以故意让模型在测试期间的前半段表现更差，然后在后半段表现更好，以获取更高的评价指标分数。**\n\n来探讨一下两种极端情况的可能性：\n- 可以省略平均 Gini 部分；\n- 可以省略关于下降率的部分。如果选择第二种情况，问题就变成了模型被推向短期表现出色但长期不稳定的情况。","metadata":{}},{"cell_type":"markdown","source":"# 提交格式\n\nFor each case_id in the test set, you must predict a probability for the target score. The file should contain a header and have the following format:\n\n```\ncase_id,score\n57543,0.1\n57544,0.9\n57545,0.5\netc.\n```","metadata":{}},{"cell_type":"markdown","source":"# 赛题时间轴\n\n- February 5, 2024 - Start Date.\n- May 20, 2024 - Entry Deadline. You must accept the competition rules before this date in order to compete.\n- May 20, 2024 - Team Merger Deadline. This is the last day participants may join or merge teams.\n- May 27, 2024 - Final Submission Deadline.","metadata":{}},{"cell_type":"markdown","source":"# 赛题数据\n\n链接: https://pan.baidu.com/s/1m6Uey6h7EMhIASqgJdFsSQ?pwd=btx6 提取码: btx6。这个比赛的数据集包含许多表格，这些表格来自各种数据源，并且在准备数据集时使用了不同级别的数据聚合。以下是数据集中的各个表格：\n\n| 数据类型 | 训练集 | 测试集 |\n| -------- | ------ | ------ |\n| Base tables         | train_base       |  test_base      |\n| static_0: internal data source         | train_static_0_0<br/>train_static_0_1       |   test_static_0_0<br/>test_static_0_1<br/>test_static_0_2     |\n| static_cb_0: external data source | train_static_cb_0 | test_static_cb_0 |\n| applprev_1: internal data source | train_applprev_1_0<br/>train_applprev_1_1 | test_applprev_1_0<br/>test_applprev_1_1<br/>test_applprev_1_2 |\n| other_1: internal data source | train_other_1 | test_other_1 |\n| tax_registry_a_1: external data source | train_tax_registry_a_1 | test_tax_registry_a_1 |\n| tax_registry_b_1: external data source | train_tax_registry_b_1 | test_tax_registry_b_1 |\n| tax_registry_c_1: external data source | train_tax_registry_c_1 | test_tax_registry_c_1 |\n| credit_bureau_a_1: external data source | train_credit_bureau_a_1_0<br/>train_credit_bureau_a_1_1<br/>train_credit_bureau_a_1_2<br/>train_credit_bureau_a_1_3 | test_credit_bureau_a_1_0<br/>test_credit_bureau_a_1_1<br/>test_credit_bureau_a_1_2<br/>test_credit_bureau_a_1_3<br/>test_credit_bureau_a_1_4 |\n| credit_bureau_b_1: external data source | train_credit_bureau_b_1 | test_credit_bureau_b_1 |\n| deposit_1: internal data source | train_deposit_1 | test_deposit_1 |\n| person_1: internal data source | train_person_1 | test_person_1 |\n| debitcard_1: internal data source | train_debitcard_1 | test_debitcard_1 |\n| applprev_2: internal data source | train_applprev_2 | test_applprev_2 |\n| person_2: internal data source | train_person_2 | test_person_2 |\n| credit_bureau_a_2: external data source | train_credit_bureau_a_2_0<br/>train_credit_bureau_a_2_1<br/>train_credit_bureau_a_2_2<br/>train_credit_bureau_a_2_3<br/>train_credit_bureau_a_2_4<br/>train_credit_bureau_a_2_5<br/>train_credit_bureau_a_2_6<br/>train_credit_bureau_a_2_7<br/>train_credit_bureau_a_2_8<br/>train_credit_bureau_a_2_9<br/>train_credit_bureau_a_2_10 | test_credit_bureau_a_2_0<br/>test_credit_bureau_a_2_1<br/>test_credit_bureau_a_2_2<br/>test_credit_bureau_a_2_3<br/>test_credit_bureau_a_2_4<br/>test_credit_bureau_a_2_5<br/>test_credit_bureau_a_2_6<br/>test_credit_bureau_a_2_7<br/>test_credit_bureau_a_2_8<br/>test_credit_bureau_a_2_9<br/>test_credit_bureau_a_2_10<br/>test_credit_bureau_a_2_11 |\n| credit_bureau_b_2: external data source | train_credit_bureau_b_2 | test_credit_bureau_b_2 |\n\n- 基础表格（Base tables）：基础表格存储了关于观察结果和`case_id`的基本信息。`case_id`是每个观察结果的唯一标识，您需要使用它将其他表格与基础表格连接起来。\n- 静态表格（Static tables）：这些表格存储了与特定`case_id`直接相关的静态特征。\n- 外部数据源表格（External data source tables）：这些表格来自外部数据源，如信用局和税务登记机构。\n- 深度1表格（Depth 1 tables）：这些表格具有深度为1的特征，每个`case_id`都有一个与之关联的历史记录。\n- 深度2表格（Depth 2 tables）：这些表格具有深度为2的特征，每个`case_id`都有一个与之关联的历史记录，同时使用了两个索引列。\n\n每个表格都有不同的列，其中一些列具有特殊含义，如`case_id`、`date_decision`、`WEEK_NUM`、`MONTH`和`target`等。除此之外，还有一些原始列用作预测变量，其定义可以在`feature_definitions.csv`文件中找到。\n\n这些表格中的数据经过了各种转换，例如转换DPD（逾期天数）、掩码类别、转换金额、转换日期等。转换过程中的各种变换由预测变量名称末尾的大写字母表示。\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd # 数据读取、数据统计\nimport numpy as np # 向量计算、矩阵\nimport os, glob\nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\n%config InlineBackend.figure_format = 'svg'\n\nDATA_PATH = '/kaggle/input/home-credit-credit-risk-model-stability/'\nos.listdir(DATA_PATH)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:08:58.686754Z","iopub.execute_input":"2024-03-11T13:08:58.687163Z","iopub.status.idle":"2024-03-11T13:08:59.600828Z","shell.execute_reply.started":"2024-03-11T13:08:58.687134Z","shell.execute_reply":"2024-03-11T13:08:59.599913Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 提交的样例\nsample_submission = pd.read_csv(os.path.join(DATA_PATH, \"sample_submission.csv\"))\nsample_submission.head(2)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:00.081629Z","iopub.execute_input":"2024-03-11T13:07:00.08232Z","iopub.status.idle":"2024-03-11T13:07:00.113276Z","shell.execute_reply.started":"2024-03-11T13:07:00.082277Z","shell.execute_reply":"2024-03-11T13:07:00.112191Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 特征/字段的含义\nfeature_definitions = pd.read_csv(os.path.join(DATA_PATH, \"feature_definitions.csv\"))\nfeature_definitions = feature_definitions.set_index('Variable')\nfeature_definitions.head(10)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:00.76852Z","iopub.execute_input":"2024-03-11T13:07:00.768881Z","iopub.status.idle":"2024-03-11T13:07:00.789245Z","shell.execute_reply.started":"2024-03-11T13:07:00.768853Z","shell.execute_reply":"2024-03-11T13:07:00.788121Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"glob.glob(os.path.join(DATA_PATH, \"parquet_files\", '*'))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:01.821845Z","iopub.execute_input":"2024-03-11T13:07:01.822333Z","iopub.status.idle":"2024-03-11T13:07:01.841151Z","shell.execute_reply.started":"2024-03-11T13:07:01.822295Z","shell.execute_reply":"2024-03-11T13:07:01.84028Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"len(glob.glob(os.path.join(DATA_PATH, \"parquet_files\", 'train', '*')))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:07.954727Z","iopub.execute_input":"2024-03-11T13:07:07.955125Z","iopub.status.idle":"2024-03-11T13:07:07.976899Z","shell.execute_reply.started":"2024-03-11T13:07:07.955093Z","shell.execute_reply":"2024-03-11T13:07:07.975876Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"len(glob.glob(os.path.join(DATA_PATH, \"parquet_files\", 'test', '*')))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:08.348755Z","iopub.execute_input":"2024-03-11T13:07:08.349121Z","iopub.status.idle":"2024-03-11T13:07:08.36128Z","shell.execute_reply.started":"2024-03-11T13:07:08.349092Z","shell.execute_reply":"2024-03-11T13:07:08.360045Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"glob.glob(os.path.join(DATA_PATH, \"parquet_files\", 'train', '*'))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:08.727575Z","iopub.execute_input":"2024-03-11T13:07:08.727917Z","iopub.status.idle":"2024-03-11T13:07:08.736641Z","shell.execute_reply.started":"2024-03-11T13:07:08.72789Z","shell.execute_reply":"2024-03-11T13:07:08.735476Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\ndef get_file_size_in_gb(file_path):\n    file_size = os.stat(file_path).st_size\n    file_size_gb = file_size / (1024 ** 3)\n    return round(file_size_gb, 3)\n\npd.DataFrame({\n    'path': [os.path.basename(x) for x in glob.glob(os.path.join(DATA_PATH, \"csv_files\", 'train', '*'))],\n    'size(gb)': [get_file_size_in_gb(x) for x in glob.glob(os.path.join(DATA_PATH, \"csv_files\", 'train', '*'))]\n}).sort_values(by='path')","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:09.275788Z","iopub.execute_input":"2024-03-11T13:07:09.276202Z","iopub.status.idle":"2024-03-11T13:07:09.311235Z","shell.execute_reply.started":"2024-03-11T13:07:09.27617Z","shell.execute_reply":"2024-03-11T13:07:09.310224Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"pd.DataFrame({\n    'path': [os.path.basename(x) for x in glob.glob(os.path.join(DATA_PATH, \"csv_files\", 'test', '*'))],\n    'size(gb)': [get_file_size_in_gb(x) for x in glob.glob(os.path.join(DATA_PATH, \"csv_files\", 'test', '*'))]\n}).sort_values(by='path')","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:09.720932Z","iopub.execute_input":"2024-03-11T13:07:09.721334Z","iopub.status.idle":"2024-03-11T13:07:09.743768Z","shell.execute_reply.started":"2024-03-11T13:07:09.721304Z","shell.execute_reply":"2024-03-11T13:07:09.742908Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# train_data = {}\n# for path in glob.glob(os.path.join(DATA_PATH, \"parquet_files\", 'train', '*')):\n#     print(path)\n#     base_name = os.path.basename(path).split('.')[0]\n#     train_data[base_name] = pd.read_parquet(path)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:19.431563Z","iopub.execute_input":"2024-03-11T13:07:19.431915Z","iopub.status.idle":"2024-03-11T13:07:19.436767Z","shell.execute_reply.started":"2024-03-11T13:07:19.431888Z","shell.execute_reply":"2024-03-11T13:07:19.435594Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\ntrain_base = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'train', 'train_base.parquet'))\ntrain_base.info()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:20.487646Z","iopub.execute_input":"2024-03-11T13:07:20.487984Z","iopub.status.idle":"2024-03-11T13:07:20.833999Z","shell.execute_reply.started":"2024-03-11T13:07:20.487958Z","shell.execute_reply":"2024-03-11T13:07:20.832865Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"%%time\n\ntrain_base = pd.read_csv(os.path.join(DATA_PATH, \"csv_files\", 'train', 'train_base.csv'))\ntrain_base.info()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:21.495506Z","iopub.execute_input":"2024-03-11T13:07:21.495893Z","iopub.status.idle":"2024-03-11T13:07:22.583395Z","shell.execute_reply.started":"2024-03-11T13:07:21.495863Z","shell.execute_reply":"2024-03-11T13:07:22.582074Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_base = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'train', 'train_base.parquet'))\ntest_base = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'test', 'test_base.parquet'))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:23.67263Z","iopub.execute_input":"2024-03-11T13:07:23.673782Z","iopub.status.idle":"2024-03-11T13:07:23.825481Z","shell.execute_reply.started":"2024-03-11T13:07:23.673732Z","shell.execute_reply":"2024-03-11T13:07:23.824582Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_base.shape, test_base.shape","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:24.434836Z","iopub.execute_input":"2024-03-11T13:07:24.435479Z","iopub.status.idle":"2024-03-11T13:07:24.444251Z","shell.execute_reply.started":"2024-03-11T13:07:24.435429Z","shell.execute_reply":"2024-03-11T13:07:24.443135Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_base.tail(2)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:24.918437Z","iopub.execute_input":"2024-03-11T13:07:24.919056Z","iopub.status.idle":"2024-03-11T13:07:24.928847Z","shell.execute_reply.started":"2024-03-11T13:07:24.919Z","shell.execute_reply":"2024-03-11T13:07:24.92776Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_base","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:25.355603Z","iopub.execute_input":"2024-03-11T13:07:25.355958Z","iopub.status.idle":"2024-03-11T13:07:25.36729Z","shell.execute_reply.started":"2024-03-11T13:07:25.355929Z","shell.execute_reply":"2024-03-11T13:07:25.366227Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 数据分析\n\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/473950\n\nhttps://www.kaggle.com/code/taichiuemura/home-credit-eda","metadata":{}},{"cell_type":"markdown","source":"## train_base / test_base","metadata":{}},{"cell_type":"code","source":"train_base['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:26.816763Z","iopub.execute_input":"2024-03-11T13:07:26.817147Z","iopub.status.idle":"2024-03-11T13:07:26.840845Z","shell.execute_reply.started":"2024-03-11T13:07:26.817117Z","shell.execute_reply":"2024-03-11T13:07:26.839549Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(8, 5))\nplt.subplot(121)\ntrain_base.groupby('MONTH')['target'].mean().plot(kind='bar')\n\nplt.subplot(122)\ntrain_base.groupby('WEEK_NUM')['target'].mean().plot()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:27.289838Z","iopub.execute_input":"2024-03-11T13:07:27.290251Z","iopub.status.idle":"2024-03-11T13:07:28.046652Z","shell.execute_reply.started":"2024-03-11T13:07:27.290219Z","shell.execute_reply":"2024-03-11T13:07:28.045508Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(8, 5))\nplt.subplot(121)\ntrain_base.groupby('MONTH')['target'].count().plot(kind='bar')\n\nplt.subplot(122)\ntrain_base.groupby('WEEK_NUM')['target'].count().plot()","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:28.04864Z","iopub.execute_input":"2024-03-11T13:07:28.049292Z","iopub.status.idle":"2024-03-11T13:07:28.763581Z","shell.execute_reply.started":"2024-03-11T13:07:28.049253Z","shell.execute_reply":"2024-03-11T13:07:28.762094Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## static_0","metadata":{}},{"cell_type":"code","source":"train_static_0_0 = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'train', 'train_static_0_0.parquet'))\ntrain_static_0_1 = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'train', 'train_static_0_1.parquet'))\n\ntest_static_0_0 = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'test', 'test_static_0_0.parquet'))\ntest_static_0_1 = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'test', 'test_static_0_1.parquet'))\ntest_static_0_2 = pd.read_parquet(os.path.join(DATA_PATH, \"parquet_files\", 'test', 'test_static_0_2.parquet'))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:29.279978Z","iopub.execute_input":"2024-03-11T13:07:29.280366Z","iopub.status.idle":"2024-03-11T13:07:34.129682Z","shell.execute_reply.started":"2024-03-11T13:07:29.280336Z","shell.execute_reply":"2024-03-11T13:07:34.128638Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_base = pd.merge(train_base, train_static_0_0, on='case_id')","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:34.131621Z","iopub.execute_input":"2024-03-11T13:07:34.131968Z","iopub.status.idle":"2024-03-11T13:07:37.784961Z","shell.execute_reply.started":"2024-03-11T13:07:34.131939Z","shell.execute_reply":"2024-03-11T13:07:37.783732Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_base.corr(numeric_only=True)['target'].abs().dropna().sort_values()","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"feature_definitions.loc['maxpmtlast3m_4525190A'].values[0]","metadata":{"execution":{"iopub.status.busy":"2024-03-11T13:07:38.705281Z","iopub.status.idle":"2024-03-11T13:07:38.70571Z","shell.execute_reply.started":"2024-03-11T13:07:38.705506Z","shell.execute_reply":"2024-03-11T13:07:38.705523Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null}]}