{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":50160,"databundleVersionId":7602123,"sourceType":"competition"},{"sourceId":162314401,"sourceType":"kernelVersion"},{"sourceId":162317063,"sourceType":"kernelVersion"},{"sourceId":162351144,"sourceType":"kernelVersion"}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Resource\n- training notebook\n  - https://www.kaggle.com/code/motono0223/home-credit-automl-training\n- packages for offline installation\n  - https://www.kaggle.com/code/motono0223/autogluon-pkgs\n  - https://www.kaggle.com/code/motono0223/ray-pkgs\n  \n# Reference \n- [1] [home-credit-baseline](https://www.kaggle.com/code/greysky/home-credit-baseline)\n- [2] [home-credit-baseline-max-min-features](https://www.kaggle.com/code/stechparme/home-credit-baseline-max-min-features)\n- [3] [dependency of autogluon (version confliction by ray package)](https://github.com/autogluon/autogluon/issues/3365)\n- [4] [Autogluon APIs](https://auto.gluon.ai/stable/api/autogluon.tabular.TabularPredictor.html)","metadata":{}},{"cell_type":"markdown","source":"> #COMENRARIOS Y MEJORAS# :\nPara mejorar el código sin alterar su funcionalidad, podrías realizar las siguientes acciones:\n\n    Optimización de la carga de datos: Verifica si hay alguna manera de optimizar la carga de datos. Por ejemplo, si la estructura de df_test es grande y se procesa repetidamente, podrías considerar almacenarla en una estructura de datos más eficiente o dividir la carga de datos en lotes más pequeños si es posible.\n\n    Mejora en el preprocesamiento de datos: Antes de realizar predicciones, podrías realizar un preprocesamiento adicional en X_test, como normalización, escalado u otra transformación de características que pueda mejorar el rendimiento del modelo.\n\n    Optimización de la predicción: Revisa si hay una forma más eficiente de hacer las predicciones utilizando el predictor. Algunas bibliotecas, como lightautoml, ofrecen opciones para mejorar el rendimiento de las predicciones mediante la configuración de parámetros o el uso de opciones de procesamiento en paralelo.\n\n    Manejo de valores faltantes: Si df_test tiene valores faltantes, podrías considerar manejarlos de manera más efectiva antes de realizar predicciones. Esto puede incluir imputación de valores faltantes o usar modelos que puedan manejar automáticamente valores faltantes.\n\n    Optimización del código: Revisa el código existente para identificar posibles cuellos de botella o áreas donde se pueda optimizar el rendimiento. Esto puede incluir la reducción de operaciones redundantes, la vectorización de operaciones si es posible y la utilización de algoritmos más eficientes.\n\n    Paralelización: Si es posible, considera paralelizar partes del proceso de predicción para aprovechar al máximo los recursos computacionales disponibles y acelerar el tiempo de ejecución.\n     Modelo actual: Si bien buscar más datos puede ser beneficioso, también es importante evaluar el modelo actual y entender por qué podría no estar funcionando tan bien como se esperaba. Esto puede implicar realizar un análisis de errores, revisar las características utilizadas en el modelo, ajustar los hiperparámetros del modelo, entre otros.\n\n    Equilibrio entre estabilidad y rendimiento: Como se menciona en el contexto, hay un equilibrio entre la estabilidad del modelo y su rendimiento. Si bien es importante buscar formas de mejorar el rendimiento del modelo, también es esencial garantizar que el modelo sea estable a lo largo del tiempo. Esto significa que el modelo debería seguir siendo efectivo incluso cuando los comportamientos de los clientes cambien.\n\n    Evaluación y ajuste continuo: Dado que los comportamientos de los clientes cambian con el tiempo, es crucial realizar evaluaciones y ajustes continuos en el modelo. Esto puede implicar actualizar regularmente el modelo con nuevos datos y realizar ajustes según sea necesario para mantener su eficacia.\n\nEn resumen, mientras buscas más datos, también es importante evaluar y ajustar continuamente tu modelo actual para garantizar que esté proporcionando predicciones precisas y estables.\n\nRecuerda siempre realizar pruebas exhaustivas después de cualquier modificación para asegurarte de que el rendimiento y la funcionalidad del código no se vean afectados negativamente. actualizado por https://www.kaggle.com/pollicio/home-credit-automl-inference-update/edit     Alejandro Cuevas V\n\n\n","metadata":{}},{"cell_type":"code","source":"!python -m pip install --no-index --find-links=/kaggle/input/autogluon-pkgs autogluon > /dev/null","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T21:20:16.572836Z","iopub.execute_input":"2024-02-11T21:20:16.573174Z","iopub.status.idle":"2024-02-11T21:23:44.051038Z","shell.execute_reply.started":"2024-02-11T21:20:16.573148Z","shell.execute_reply":"2024-02-11T21:23:44.049981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python -m pip install --no-index --find-links=/kaggle/input/ray-pkgs --upgrade --force-reinstall -q ray==2.6.3","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T21:23:44.052984Z","iopub.execute_input":"2024-02-11T21:23:44.053278Z","iopub.status.idle":"2024-02-11T21:24:14.911681Z","shell.execute_reply.started":"2024-02-11T21:23:44.053250Z","shell.execute_reply":"2024-02-11T21:24:14.910614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport gc\nfrom glob import glob\nfrom pathlib import Path\nfrom datetime import datetime\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.model_selection import StratifiedGroupKFold\nfrom sklearn.base import BaseEstimator, RegressorMixin\nimport joblib\nimport lightgbm as lgb\nimport warnings\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\nfrom autogluon.tabular import TabularDataset, TabularPredictor","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:14.912972Z","iopub.execute_input":"2024-02-11T21:24:14.913271Z","iopub.status.idle":"2024-02-11T21:24:20.143499Z","shell.execute_reply.started":"2024-02-11T21:24:14.913243Z","shell.execute_reply":"2024-02-11T21:24:20.142661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Pipeline","metadata":{}},{"cell_type":"code","source":"class Pipeline:\n    @staticmethod\n    def set_table_dtypes(df): #Standardize the dtype.\n        for col in df.columns:\n            if col in [\"case_id\", \"WEEK_NUM\", \"num_group1\", \"num_group2\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Int64))\n            elif col in [\"date_decision\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Date))\n            elif col[-1] in (\"P\", \"A\"):\n                df = df.with_columns(pl.col(col).cast(pl.Float64))\n            elif col[-1] in (\"M\",):\n                df = df.with_columns(pl.col(col).cast(pl.String))\n            elif col[-1] in (\"D\",):\n                df = df.with_columns(pl.col(col).cast(pl.Date))            \n\n        return df\n    \n    @staticmethod\n    def handle_dates(df): #Change the feature for D to the difference in days from date_decision.\n        for col in df.columns:\n            if col[-1] in (\"D\",):\n                df = df.with_columns(pl.col(col) - pl.col(\"date_decision\"))\n                df = df.with_columns(pl.col(col).dt.total_days())\n                \n        df = df.drop(\"date_decision\", \"MONTH\")\n\n        return df\n    \n    @staticmethod\n    def filter_cols(df): #Remove those with an average is_null exceeding 0.95 and those that do not fall within the range 1 < nunique < 200.\n        for col in df.columns:\n            if col not in [\"target\", \"case_id\", \"WEEK_NUM\"]:\n                isnull = df[col].is_null().mean()\n\n                if isnull > 0.95:\n                    df = df.drop(col)\n\n        for col in df.columns:\n            if (col not in [\"target\", \"case_id\", \"WEEK_NUM\"]) & (df[col].dtype == pl.String):\n                freq = df[col].n_unique()\n\n                if (freq == 1) | (freq > 200):\n                    df = df.drop(col)\n\n        return df","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.145980Z","iopub.execute_input":"2024-02-11T21:24:20.146633Z","iopub.status.idle":"2024-02-11T21:24:20.159796Z","shell.execute_reply.started":"2024-02-11T21:24:20.146599Z","shell.execute_reply":"2024-02-11T21:24:20.158933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Automatic Aggregation","metadata":{}},{"cell_type":"code","source":"class Aggregator:\n    @staticmethod\n    def num_expr(df): #Extract the maximum and minimum values for features P and A, and add them as additional features.\n        cols = [col for col in df.columns if col[-1] in (\"P\", \"A\")]\n\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n\n        return expr_max, expr_min\n\n    @staticmethod\n    def date_expr(df): #Extract the maximum and minimum values for features D, and add them as additional features.\n        cols = [col for col in df.columns if col[-1] in (\"D\",)]\n\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n\n        return expr_max, expr_min\n\n    @staticmethod\n    def str_expr(df): #Extract the maximum and minimum values for features M, and add them as additional features.\n        cols = [col for col in df.columns if col[-1] in (\"M\",)]\n        \n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n\n        return expr_max, expr_min\n\n    @staticmethod\n    def other_expr(df): #Extract the maximum and minimum values for features T and L, and add them as additional features.\n        cols = [col for col in df.columns if col[-1] in (\"T\", \"L\")]\n        \n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n\n        return expr_max, expr_min\n    \n    @staticmethod\n    def count_expr(df): #Extract the maximum and minimum values for each num_group and add them as additional features.\n        cols = [col for col in df.columns if \"num_group\" in col]\n\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n\n        return expr_max, expr_min\n\n    @staticmethod\n    def get_exprs(df): #Execute the above function and return the result.\n        maxexprs = Aggregator.num_expr(df)[0] + \\\n                Aggregator.date_expr(df)[0] + \\\n                Aggregator.str_expr(df)[0] + \\\n                Aggregator.other_expr(df)[0] + \\\n                Aggregator.count_expr(df)[0]\n        \n        minexprs = Aggregator.num_expr(df)[1] + \\\n                Aggregator.date_expr(df)[1] + \\\n                Aggregator.str_expr(df)[1] + \\\n                Aggregator.other_expr(df)[1] + \\\n                Aggregator.count_expr(df)[1]\n\n        return maxexprs, minexprs","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.161045Z","iopub.execute_input":"2024-02-11T21:24:20.161384Z","iopub.status.idle":"2024-02-11T21:24:20.179783Z","shell.execute_reply.started":"2024-02-11T21:24:20.161354Z","shell.execute_reply":"2024-02-11T21:24:20.178877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# File I/O","metadata":{}},{"cell_type":"code","source":"def read_file(path, depth=None): \n    df = pl.read_parquet(path)\n    df = df.pipe(Pipeline.set_table_dtypes)\n    \n    if depth in [1, 2]:\n        maxexprs, minexprs = Aggregator.get_exprs(df)\n        df = df.group_by(\"case_id\").agg(*maxexprs, *minexprs)\n    \n    return df\n\ndef read_files(regex_path, depth=None):\n    chunks = []\n    for path in glob(str(regex_path)):\n        chunks.append(pl.read_parquet(path).pipe(Pipeline.set_table_dtypes))\n        \n    df = pl.concat(chunks, how=\"vertical_relaxed\")\n    if depth in [1, 2]:\n        maxexprs, minexprs = Aggregator.get_exprs(df)\n        df = df.group_by(\"case_id\").agg(*maxexprs, *minexprs)\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.180912Z","iopub.execute_input":"2024-02-11T21:24:20.181204Z","iopub.status.idle":"2024-02-11T21:24:20.194364Z","shell.execute_reply.started":"2024-02-11T21:24:20.181164Z","shell.execute_reply":"2024-02-11T21:24:20.193449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"code","source":"def feature_eng(df_base, depth_0, depth_1, depth_2):\n    df_base = (\n        df_base\n        .with_columns(\n            month_decision = pl.col(\"date_decision\").dt.month(),\n            weekday_decision = pl.col(\"date_decision\").dt.weekday(),\n        )\n    )\n        \n    for i, df in enumerate(depth_0 + depth_1 + depth_2):\n        df_base = df_base.join(df, how=\"left\", on=\"case_id\", suffix=f\"_{i}\")\n        \n    df_base = df_base.pipe(Pipeline.handle_dates)\n    \n    return df_base","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.195636Z","iopub.execute_input":"2024-02-11T21:24:20.195968Z","iopub.status.idle":"2024-02-11T21:24:20.205175Z","shell.execute_reply.started":"2024-02-11T21:24:20.195937Z","shell.execute_reply":"2024-02-11T21:24:20.204364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def to_pandas(df_data, cat_cols=None):\n    df_data = df_data.to_pandas()\n    \n    if cat_cols is None:\n        cat_cols = list(df_data.select_dtypes(\"object\").columns)\n    \n    df_data[cat_cols] = df_data[cat_cols].astype(\"category\")\n    \n    return df_data, cat_cols","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.206391Z","iopub.execute_input":"2024-02-11T21:24:20.206710Z","iopub.status.idle":"2024-02-11T21:24:20.216083Z","shell.execute_reply.started":"2024-02-11T21:24:20.206687Z","shell.execute_reply":"2024-02-11T21:24:20.215327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        if str(col_type)==\"category\":\n            continue\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            df[col] = df[col].astype('category')\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.217308Z","iopub.execute_input":"2024-02-11T21:24:20.217887Z","iopub.status.idle":"2024-02-11T21:24:20.230683Z","shell.execute_reply.started":"2024-02-11T21:24:20.217856Z","shell.execute_reply":"2024-02-11T21:24:20.229879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration","metadata":{}},{"cell_type":"code","source":"sample = pd.read_csv(\"/kaggle/input/home-credit-credit-risk-model-stability/sample_submission.csv\")\nDRY_RUN = True if sample.shape[0] == 10 else False   # if num of records of test data is 10, dry-run is enable.\nPRESETS = \"medium_quality\"\nMODEL_PATH = \"/kaggle/input/home-credit-automl-training/predictor\"","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.235289Z","iopub.execute_input":"2024-02-11T21:24:20.235621Z","iopub.status.idle":"2024-02-11T21:24:20.253750Z","shell.execute_reply.started":"2024-02-11T21:24:20.235600Z","shell.execute_reply":"2024-02-11T21:24:20.252973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT            = Path(\"/kaggle/input/home-credit-credit-risk-model-stability\")\nTRAIN_DIR       = ROOT / \"parquet_files\" / \"train\"\nTEST_DIR        = ROOT / \"parquet_files\" / \"test\"","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.255098Z","iopub.execute_input":"2024-02-11T21:24:20.255450Z","iopub.status.idle":"2024-02-11T21:24:20.260077Z","shell.execute_reply.started":"2024-02-11T21:24:20.255420Z","shell.execute_reply":"2024-02-11T21:24:20.258996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Files Read & Feature Engineering","metadata":{}},{"cell_type":"code","source":"data_store = {\n    \"df_base\": read_file(TRAIN_DIR / \"train_base.parquet\"),\n    \"depth_0\": [\n        read_file(TRAIN_DIR / \"train_static_cb_0.parquet\"),\n        read_files(TRAIN_DIR / \"train_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        read_files(TRAIN_DIR / \"train_applprev_1_*.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_a_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_b_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_c_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_other_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_person_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_deposit_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        read_file(TRAIN_DIR / \"train_credit_bureau_b_2.parquet\", 2),\n    ]\n}","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:20.261338Z","iopub.execute_input":"2024-02-11T21:24:20.261627Z","iopub.status.idle":"2024-02-11T21:24:54.918687Z","shell.execute_reply.started":"2024-02-11T21:24:20.261604Z","shell.execute_reply":"2024-02-11T21:24:54.917705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = feature_eng(**data_store)\nprint(\"train data shape:\\t\", df_train.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:24:54.920159Z","iopub.execute_input":"2024-02-11T21:24:54.920551Z","iopub.status.idle":"2024-02-11T21:25:06.149773Z","shell.execute_reply.started":"2024-02-11T21:24:54.920515Z","shell.execute_reply":"2024-02-11T21:25:06.148856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = df_train.pipe(Pipeline.filter_cols)\nprint(\"train data shape:\\t\", df_train.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:06.151197Z","iopub.execute_input":"2024-02-11T21:25:06.151498Z","iopub.status.idle":"2024-02-11T21:25:09.581941Z","shell.execute_reply.started":"2024-02-11T21:25:06.151473Z","shell.execute_reply":"2024-02-11T21:25:09.580961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train, cat_cols = to_pandas(df_train)\nprint(df_train.shape)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:09.583064Z","iopub.execute_input":"2024-02-11T21:25:09.583343Z","iopub.status.idle":"2024-02-11T21:25:30.992451Z","shell.execute_reply.started":"2024-02-11T21:25:09.583318Z","shell.execute_reply":"2024-02-11T21:25:30.991525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del data_store\ndf_train = reduce_mem_usage(df_train)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:30.993762Z","iopub.execute_input":"2024-02-11T21:25:30.994068Z","iopub.status.idle":"2024-02-11T21:25:37.003655Z","shell.execute_reply.started":"2024-02-11T21:25:30.994042Z","shell.execute_reply":"2024-02-11T21:25:37.002638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"if DRY_RUN:\n    print(f\"df_train.shape : {df_train.shape} --> \", end=\"\")\n    df_train = df_train.iloc[:500]\n    print( df_train.shape )","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.004939Z","iopub.execute_input":"2024-02-11T21:25:37.005263Z","iopub.status.idle":"2024-02-11T21:25:37.011500Z","shell.execute_reply.started":"2024-02-11T21:25:37.005236Z","shell.execute_reply":"2024-02-11T21:25:37.010600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictor = TabularPredictor(\n   label=\"target\",\n   problem_type=\"binary\",\n   eval_metric=\"roc_auc\",\n   path=\"predictor\",\n)\n\n# Load model\npredictor = TabularPredictor.load(path=MODEL_PATH)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.012625Z","iopub.execute_input":"2024-02-11T21:25:37.012888Z","iopub.status.idle":"2024-02-11T21:25:37.508833Z","shell.execute_reply.started":"2024-02-11T21:25:37.012865Z","shell.execute_reply":"2024-02-11T21:25:37.507847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#weeks = df_train[\"WEEK_NUM\"]\n#df_train = df_train.drop(columns=[\"case_id\", \"WEEK_NUM\"])\n#cv = StratifiedGroupKFold(n_splits=5, shuffle=False)\n#for idx_train, idx_valid in cv.split(df_train, df_train[\"target\"], groups=weeks):\n    #fold_train = df_train.iloc[idx_train]\n    #fold_valid = df_train.iloc[idx_valid]\n#train_data = TabularDataset(fold_train)\n#valid_data = TabularDataset(fold_valid)\n           ","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.510519Z","iopub.execute_input":"2024-02-11T21:25:37.511202Z","iopub.status.idle":"2024-02-11T21:25:37.515384Z","shell.execute_reply.started":"2024-02-11T21:25:37.511171Z","shell.execute_reply":"2024-02-11T21:25:37.514324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#del df_train # It is deleted to reduce memory usage\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.516814Z","iopub.execute_input":"2024-02-11T21:25:37.517179Z","iopub.status.idle":"2024-02-11T21:25:37.652650Z","shell.execute_reply.started":"2024-02-11T21:25:37.517144Z","shell.execute_reply":"2024-02-11T21:25:37.651578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#%%time\n#predictor.fit(\n#    train_data,\n#    tuning_data=valid_data,\n#    save_space=True,\n#    presets=PRESETS,\n#    use_bag_holdout=True,\n#    ag_args_fit={'num_gpus': 1},\n#)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-11T21:25:37.653845Z","iopub.execute_input":"2024-02-11T21:25:37.654136Z","iopub.status.idle":"2024-02-11T21:25:37.661176Z","shell.execute_reply.started":"2024-02-11T21:25:37.654112Z","shell.execute_reply":"2024-02-11T21:25:37.660331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#gc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.662278Z","iopub.execute_input":"2024-02-11T21:25:37.662546Z","iopub.status.idle":"2024-02-11T21:25:37.669721Z","shell.execute_reply.started":"2024-02-11T21:25:37.662523Z","shell.execute_reply":"2024-02-11T21:25:37.668899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training result","metadata":{}},{"cell_type":"code","source":"predictor.leaderboard()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.670716Z","iopub.execute_input":"2024-02-11T21:25:37.670967Z","iopub.status.idle":"2024-02-11T21:25:37.702059Z","shell.execute_reply.started":"2024-02-11T21:25:37.670946Z","shell.execute_reply":"2024-02-11T21:25:37.701143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_lb = predictor.leaderboard()\nfrom matplotlib import pyplot as plt\nplt.scatter( df_lb[\"score_val\"], df_lb[\"model\"] )\nplt.grid()\nplt.xlabel(\"CV(roc_auc)\")\nplt.ylabel(\"Model name\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-11T21:25:37.703470Z","iopub.execute_input":"2024-02-11T21:25:37.704149Z","iopub.status.idle":"2024-02-11T21:25:37.957154Z","shell.execute_reply.started":"2024-02-11T21:25:37.704112Z","shell.execute_reply":"2024-02-11T21:25:37.956256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test Files Read & Feature Engineering","metadata":{}},{"cell_type":"code","source":"data_store = {\n    \"df_base\": read_file(TEST_DIR / \"test_base.parquet\"),\n    \"depth_0\": [\n        read_file(TEST_DIR / \"test_static_cb_0.parquet\"),\n        read_files(TEST_DIR / \"test_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        read_files(TEST_DIR / \"test_applprev_1_*.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_a_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_b_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_c_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_credit_bureau_b_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_other_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_person_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_deposit_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        read_file(TEST_DIR / \"test_credit_bureau_b_2.parquet\", 2),\n    ]\n}","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:37.958504Z","iopub.execute_input":"2024-02-11T21:25:37.959172Z","iopub.status.idle":"2024-02-11T21:25:38.148271Z","shell.execute_reply.started":"2024-02-11T21:25:37.959136Z","shell.execute_reply":"2024-02-11T21:25:38.147164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = feature_eng(**data_store)\nprint(\"test data shape:\\t\", df_test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.149824Z","iopub.execute_input":"2024-02-11T21:25:38.150589Z","iopub.status.idle":"2024-02-11T21:25:38.193235Z","shell.execute_reply.started":"2024-02-11T21:25:38.150552Z","shell.execute_reply":"2024-02-11T21:25:38.192292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = df_test.select([col for col in df_test.columns if col != \"target\"])\nprint(\"test data shape:\\t\", df_test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.194318Z","iopub.execute_input":"2024-02-11T21:25:38.194613Z","iopub.status.idle":"2024-02-11T21:25:38.202987Z","shell.execute_reply.started":"2024-02-11T21:25:38.194588Z","shell.execute_reply":"2024-02-11T21:25:38.202030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test, cat_cols = to_pandas(df_test, cat_cols) # cat_cols was created by train data","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.207985Z","iopub.execute_input":"2024-02-11T21:25:38.208307Z","iopub.status.idle":"2024-02-11T21:25:38.277262Z","shell.execute_reply.started":"2024-02-11T21:25:38.208270Z","shell.execute_reply":"2024-02-11T21:25:38.276456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_test.shape)\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.278642Z","iopub.execute_input":"2024-02-11T21:25:38.278916Z","iopub.status.idle":"2024-02-11T21:25:38.304598Z","shell.execute_reply.started":"2024-02-11T21:25:38.278892Z","shell.execute_reply":"2024-02-11T21:25:38.303537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Considerar la inclusión de más información: Si es relevante, podrías mostrar el número total de valores nulos encontrados en la columna \"score\".","metadata":{}},{"cell_type":"code","source":" if \"assignmentdate_238D\" in df_test.columns:\n    null_count = df_test[\"assignmentdate_238D\"].isnull().sum()\n    if null_count > 0:\n        print(f\"Total null values in the 'assignmentdate_238D' column: {null_count}\")\nelse:\n    print(\"La columna 'assignmentdate_238D' no está presente en el DataFrame df_test.\")","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.305903Z","iopub.execute_input":"2024-02-11T21:25:38.306253Z","iopub.status.idle":"2024-02-11T21:25:38.314586Z","shell.execute_reply.started":"2024-02-11T21:25:38.306226Z","shell.execute_reply":"2024-02-11T21:25:38.313664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(df_test.shape)\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.315649Z","iopub.execute_input":"2024-02-11T21:25:38.316000Z","iopub.status.idle":"2024-02-11T21:25:38.347513Z","shell.execute_reply.started":"2024-02-11T21:25:38.315972Z","shell.execute_reply":"2024-02-11T21:25:38.346524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"median_value = df_test[\"assignmentdate_4955616D\"].median()\ndf_test[\"assignmentdate_4955616D\"].fillna(median_value, inplace=True)\n\nmedian_value = df_test[\"contractssum_5085716L\"].median()\ndf_test[\"contractssum_5085716L\"].fillna(median_value, inplace=True)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.348813Z","iopub.execute_input":"2024-02-11T21:25:38.349186Z","iopub.status.idle":"2024-02-11T21:25:38.357264Z","shell.execute_reply.started":"2024-02-11T21:25:38.349151Z","shell.execute_reply":"2024-02-11T21:25:38.356344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lista de columnas con valores nulos\ncolumns_with_nulls = ['max_birth_259D', 'max_num_group1_8',\"max_pmts_dpdvalue_108P\",\"birthdate_574D\",\"assignmentdate_4527235D\",\"assignmentdate_238D\"]\n\n# Imputar valores nulos con 0 en las columnas específicas\ndf_test[columns_with_nulls] = df_test[columns_with_nulls].fillna(0)\n\n# Verificar si hay valores nulos después de la imputación\nnull_count_after_imputation = df_test[columns_with_nulls].isnull().sum()\n\n# Mostrar el conteo de valores nulos después de la imputación\nprint(\"Valores nulos después de la imputación:\")\nprint(null_count_after_imputation)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.358384Z","iopub.execute_input":"2024-02-11T21:25:38.358699Z","iopub.status.idle":"2024-02-11T21:25:38.371729Z","shell.execute_reply.started":"2024-02-11T21:25:38.358676Z","shell.execute_reply":"2024-02-11T21:25:38.370739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#del data_store\ndf_test = reduce_mem_usage(df_test)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.372770Z","iopub.execute_input":"2024-02-11T21:25:38.373049Z","iopub.status.idle":"2024-02-11T21:25:38.756207Z","shell.execute_reply.started":"2024-02-11T21:25:38.373025Z","shell.execute_reply":"2024-02-11T21:25:38.755269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction","metadata":{}},{"cell_type":"code","source":"X_test = df_test.drop(columns=[\"case_id\", \"WEEK_NUM\"])\ntest_data = TabularDataset(df_test)\ny_pred = predictor.predict_proba(test_data).iloc[:, 1].values","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:38.757190Z","iopub.execute_input":"2024-02-11T21:25:38.757488Z","iopub.status.idle":"2024-02-11T21:25:52.104793Z","shell.execute_reply.started":"2024-02-11T21:25:38.757464Z","shell.execute_reply":"2024-02-11T21:25:52.103900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"df_subm = pd.read_csv(ROOT / \"sample_submission.csv\")\ndf_subm = df_subm.set_index(\"case_id\")\n\ndf_subm[\"score\"] = y_pred","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:52.106055Z","iopub.execute_input":"2024-02-11T21:25:52.106720Z","iopub.status.idle":"2024-02-11T21:25:52.116140Z","shell.execute_reply.started":"2024-02-11T21:25:52.106690Z","shell.execute_reply":"2024-02-11T21:25:52.115055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Check null: \", df_subm[\"score\"].isnull().any())\n\ndf_subm.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:52.117642Z","iopub.execute_input":"2024-02-11T21:25:52.118092Z","iopub.status.idle":"2024-02-11T21:25:52.138762Z","shell.execute_reply.started":"2024-02-11T21:25:52.118054Z","shell.execute_reply":"2024-02-11T21:25:52.137582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Verificar si hay valores nulos en la columna \"score\" del DataFrame df_subm\nprint(\"Check null values in the 'score' column: \", df_subm[\"score\"].isnull().any())\n\n# Mostrar las primeras filas del DataFrame df_subm\ndf_subm.head()\n","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:52.140111Z","iopub.execute_input":"2024-02-11T21:25:52.140487Z","iopub.status.idle":"2024-02-11T21:25:52.153492Z","shell.execute_reply.started":"2024-02-11T21:25:52.140459Z","shell.execute_reply":"2024-02-11T21:25:52.152366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_subm.to_csv(\"submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-02-11T21:25:52.155000Z","iopub.execute_input":"2024-02-11T21:25:52.155370Z","iopub.status.idle":"2024-02-11T21:25:52.168482Z","shell.execute_reply.started":"2024-02-11T21:25:52.155336Z","shell.execute_reply":"2024-02-11T21:25:52.167578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n   ","metadata":{}}]}