{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30699,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"papermill":{"default_parameters":{},"duration":2840.363124,"end_time":"2024-05-08T14:46:48.044187","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-05-08T13:59:27.681063","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HOME CREDIT WITH AVERAGE ENSEMBLE MODEL 🏡\n\n<font color = 'green'>\n    Contents:\n\n1. [Load and Read Data](#1)\n    \n2. [Check Data](#2)\n    \n3. [Preprocessing](#3)\n    \n4. [Encode Categorical Features](#4)\n    \n5. [Modelling](#5)\n    \n    5.1. [Feature Importance](#6)\n    \n    5.2. [Optuna Hyperparameter Tunning for LGBM](#7)\n    \n    5.3. [Cross Validation](#8)\n    \n    5.4. [Average Ensemble Model](#9)\n    \n    5.5. [Stacking Model](#10)\n    \n6. [Submission (for Average Ensemble Model)](#11)","metadata":{"papermill":{"duration":0.019165,"end_time":"2024-05-08T13:59:30.801539","exception":false,"start_time":"2024-05-08T13:59:30.782374","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nimport gc\nfrom glob import glob\nfrom pathlib import Path\nfrom datetime import datetime\nimport numpy as np\nimport pandas as pd\nimport polars as pl\n\nfrom sklearn.preprocessing import LabelEncoder\n\npd.options.display.max_columns=1000\npd.options.display.max_rows=1000\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_palette(\"Pastel1\")\n\nimport warnings as wr\nwr.filterwarnings('ignore')","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":2.433225,"end_time":"2024-05-08T13:59:33.253088","exception":false,"start_time":"2024-05-08T13:59:30.819863","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:51.871745Z","iopub.execute_input":"2024-05-09T15:10:51.872217Z","iopub.status.idle":"2024-05-09T15:10:52.969656Z","shell.execute_reply.started":"2024-05-09T15:10:51.872142Z","shell.execute_reply":"2024-05-09T15:10:52.968541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Load and Read Data","metadata":{"papermill":{"duration":0.017155,"end_time":"2024-05-08T13:59:33.288083","exception":false,"start_time":"2024-05-08T13:59:33.270928","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class Pipeline:\n    @staticmethod\n    def set_table_dtypes(df):\n        for col in df.columns:\n            if col in [\"case_id\", \"WEEK_NUM\", \"num_group1\", \"num_group2\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Int32))\n            elif col in [\"date_decision\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Date))\n            elif col[-1] in (\"P\", \"A\"):\n                df = df.with_columns(pl.col(col).cast(pl.Float64))\n            elif col[-1] in (\"M\",):\n                df = df.with_columns(pl.col(col).cast(pl.String))\n            elif col[-1] in (\"D\",):\n                df = df.with_columns(pl.col(col).cast(pl.Date))            \n\n        return df\n    \n    @staticmethod\n    def handle_dates(df):\n        for col in df.columns:\n            if col[-1] in (\"D\",):\n                df = df.with_columns(pl.col(col) - pl.col(\"date_decision\"))\n                df = df.with_columns(pl.col(col).dt.total_days())\n                df = df.with_columns(pl.col(col).cast(pl.Float32))\n                \n        df = df.drop(\"date_decision\", \"MONTH\")\n\n        return df\n    \n    @staticmethod\n    def filter_cols(df):\n        for col in df.columns:\n            if col not in [\"target\", \"case_id\", \"WEEK_NUM\"]:\n                isnull = df[col].is_null().mean()\n\n                if isnull > 0.95:\n                    df = df.drop(col)\n\n        for col in df.columns:\n            if (col not in [\"target\", \"case_id\", \"WEEK_NUM\"]) & (df[col].dtype == pl.String):\n                freq = df[col].n_unique()\n\n                if (freq == 1) | (freq > 200):\n                    df = df.drop(col)\n\n        return df","metadata":{"papermill":{"duration":0.034819,"end_time":"2024-05-08T13:59:33.341103","exception":false,"start_time":"2024-05-08T13:59:33.306284","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:52.975358Z","iopub.execute_input":"2024-05-09T15:10:52.975666Z","iopub.status.idle":"2024-05-09T15:10:52.989690Z","shell.execute_reply.started":"2024-05-09T15:10:52.975638Z","shell.execute_reply":"2024-05-09T15:10:52.988604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Aggregator:\n    @staticmethod\n    def num_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"P\", \"A\")]\n\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n\n        return expr_max\n\n    @staticmethod\n    def date_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"D\",)]\n\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n\n        return expr_max\n\n    @staticmethod\n    def str_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"M\",)]\n        \n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n\n        return expr_max\n\n    @staticmethod\n    def other_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"T\", \"L\")]\n        \n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n\n        return expr_max\n    \n    @staticmethod\n    def count_expr(df):\n        cols = [col for col in df.columns if \"num_group\" in col]\n\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n\n        return expr_max\n\n    @staticmethod\n    def get_exprs(df):\n        exprs = Aggregator.num_expr(df) + \\\n                Aggregator.date_expr(df) + \\\n                Aggregator.str_expr(df) + \\\n                Aggregator.other_expr(df) + \\\n                Aggregator.count_expr(df)\n\n        return exprs","metadata":{"papermill":{"duration":0.031033,"end_time":"2024-05-08T13:59:33.389208","exception":false,"start_time":"2024-05-08T13:59:33.358175","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:52.991024Z","iopub.execute_input":"2024-05-09T15:10:52.991354Z","iopub.status.idle":"2024-05-09T15:10:53.007394Z","shell.execute_reply.started":"2024-05-09T15:10:52.991323Z","shell.execute_reply":"2024-05-09T15:10:53.006541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_file(path, depth=None):\n    df = pl.read_parquet(path)\n    df = df.pipe(Pipeline.set_table_dtypes)\n    \n    if depth in [1, 2]:\n        df = df.group_by(\"case_id\").agg(Aggregator.get_exprs(df))\n    \n    return df\n\ndef read_files(regex_path, depth=None):\n    chunks = []\n    for path in glob(str(regex_path)):\n        df = pl.read_parquet(path)\n        df = df.pipe(Pipeline.set_table_dtypes)\n        \n        if depth in [1, 2]:\n            df = df.group_by(\"case_id\").agg(Aggregator.get_exprs(df))\n        \n        chunks.append(df)\n        \n    df = pl.concat(chunks, how=\"vertical_relaxed\")\n    df = df.unique(subset=[\"case_id\"])\n    \n    return df","metadata":{"papermill":{"duration":0.027826,"end_time":"2024-05-08T13:59:33.433183","exception":false,"start_time":"2024-05-08T13:59:33.405357","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:53.010901Z","iopub.execute_input":"2024-05-09T15:10:53.011172Z","iopub.status.idle":"2024-05-09T15:10:53.022317Z","shell.execute_reply.started":"2024-05-09T15:10:53.011148Z","shell.execute_reply":"2024-05-09T15:10:53.021426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def feature_eng(df_base, depth_0, depth_1, depth_2):\n    df_base = (\n        df_base\n        .with_columns(\n            month_decision = pl.col(\"date_decision\").dt.month(),\n            weekday_decision = pl.col(\"date_decision\").dt.weekday(),\n        )\n    )\n        \n    for i, df in enumerate(depth_0 + depth_1 + depth_2):\n        df_base = df_base.join(df, how=\"left\", on=\"case_id\", suffix=f\"_{i}\")\n        \n    df_base = df_base.pipe(Pipeline.handle_dates)\n    \n    return df_base","metadata":{"papermill":{"duration":0.026515,"end_time":"2024-05-08T13:59:33.476855","exception":false,"start_time":"2024-05-08T13:59:33.450340","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:53.023581Z","iopub.execute_input":"2024-05-09T15:10:53.024451Z","iopub.status.idle":"2024-05-09T15:10:53.033796Z","shell.execute_reply.started":"2024-05-09T15:10:53.024423Z","shell.execute_reply":"2024-05-09T15:10:53.032759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def to_pandas(df_data, cat_cols=None):\n    df_data = df_data.to_pandas()\n    if cat_cols is None:\n        cat_cols = list(df_data.select_dtypes(\"object\").columns)\n    df_data[cat_cols] = df_data[cat_cols].astype(\"category\")\n    return df_data, cat_cols","metadata":{"papermill":{"duration":0.025703,"end_time":"2024-05-08T13:59:33.519731","exception":false,"start_time":"2024-05-08T13:59:33.494028","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:53.034880Z","iopub.execute_input":"2024-05-09T15:10:53.035151Z","iopub.status.idle":"2024-05-09T15:10:53.048385Z","shell.execute_reply.started":"2024-05-09T15:10:53.035127Z","shell.execute_reply":"2024-05-09T15:10:53.047577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        if str(col_type)==\"category\":\n            continue\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            continue\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","metadata":{"_kg_hide-output":true,"papermill":{"duration":0.033298,"end_time":"2024-05-08T13:59:33.569935","exception":false,"start_time":"2024-05-08T13:59:33.536637","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:53.049424Z","iopub.execute_input":"2024-05-09T15:10:53.049704Z","iopub.status.idle":"2024-05-09T15:10:53.064496Z","shell.execute_reply.started":"2024-05-09T15:10:53.049679Z","shell.execute_reply":"2024-05-09T15:10:53.063593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ROOT            = Path(\"/kaggle/input/home-credit-credit-risk-model-stability\")\nTRAIN_DIR       = ROOT / \"parquet_files\" / \"train\"\nTEST_DIR        = ROOT / \"parquet_files\" / \"test\"","metadata":{"papermill":{"duration":0.024331,"end_time":"2024-05-08T13:59:33.611145","exception":false,"start_time":"2024-05-08T13:59:33.586814","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:53.065814Z","iopub.execute_input":"2024-05-09T15:10:53.066182Z","iopub.status.idle":"2024-05-09T15:10:53.081012Z","shell.execute_reply.started":"2024-05-09T15:10:53.066147Z","shell.execute_reply":"2024-05-09T15:10:53.080208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train","metadata":{"papermill":{"duration":0.016635,"end_time":"2024-05-08T13:59:33.644779","exception":false,"start_time":"2024-05-08T13:59:33.628144","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_store = {\n    \"df_base\": read_file(TRAIN_DIR / \"train_base.parquet\"),\n    \"depth_0\": [\n        read_file(TRAIN_DIR / \"train_static_cb_0.parquet\"),\n        read_files(TRAIN_DIR / \"train_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        read_files(TRAIN_DIR / \"train_applprev_1_*.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_a_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_b_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_c_1.parquet\", 1),\n        read_files(TRAIN_DIR / \"train_credit_bureau_a_1_*.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_other_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_person_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_deposit_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        read_file(TRAIN_DIR / \"train_credit_bureau_b_2.parquet\", 2),\n        read_files(TRAIN_DIR / \"train_credit_bureau_a_2_*.parquet\", 2),\n    ]\n}","metadata":{"papermill":{"duration":139.821456,"end_time":"2024-05-08T14:01:53.482978","exception":false,"start_time":"2024-05-08T13:59:33.661522","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:10:53.082237Z","iopub.execute_input":"2024-05-09T15:10:53.082588Z","iopub.status.idle":"2024-05-09T15:12:59.966498Z","shell.execute_reply.started":"2024-05-09T15:10:53.082540Z","shell.execute_reply":"2024-05-09T15:12:59.965694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = feature_eng(**data_store)\nprint(\"train data shape:\\t\", train_df.shape)","metadata":{"papermill":{"duration":10.714414,"end_time":"2024-05-08T14:02:04.214346","exception":false,"start_time":"2024-05-08T14:01:53.499932","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:12:59.967823Z","iopub.execute_input":"2024-05-09T15:12:59.968485Z","iopub.status.idle":"2024-05-09T15:13:10.600455Z","shell.execute_reply.started":"2024-05-09T15:12:59.968447Z","shell.execute_reply":"2024-05-09T15:13:10.599472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndel data_store\ngc.collect()\ntrain_df = train_df.pipe(Pipeline.filter_cols)\ntrain_df, cat_cols = to_pandas(train_df)\ntrain_df = reduce_mem_usage(train_df)\nprint(\"train data shape:\\t\", train_df.shape)\nnums=train_df.select_dtypes(exclude='category').columns\nfrom itertools import combinations, permutations\n#train_df=train_df[nums]\nnans_df = train_df[nums].isna()\nnans_groups={}\nfor col in nums:\n    cur_group = nans_df[col].sum()\n    try:\n        nans_groups[cur_group].append(col)\n    except:\n        nans_groups[cur_group]=[col]\ndel nans_df; x=gc.collect()\n\ndef reduce_group(grps):\n    use = []\n    for g in grps:\n        mx = 0; vx = g[0]\n        for gg in g:\n            n = train_df[gg].nunique()\n            if n>mx:\n                mx = n\n                vx = gg\n            #print(str(gg)+'-'+str(n),', ',end='')\n        use.append(vx)\n        #print()\n    print('Use these',use)\n    return use\n\ndef group_columns_by_correlation(matrix, threshold=0.8):\n    \n    correlation_matrix = matrix.corr()\n    groups = []\n    remaining_cols = list(matrix.columns)\n    while remaining_cols:\n        col = remaining_cols.pop(0)\n        group = [col]\n        correlated_cols = [col]\n        for c in remaining_cols:\n            if correlation_matrix.loc[col, c] >= threshold:\n                group.append(c)\n                correlated_cols.append(c)\n        groups.append(group)\n        remaining_cols = [c for c in remaining_cols if c not in correlated_cols]\n    \n    return groups\n\nuses=[]\nfor k,v in nans_groups.items():\n    if len(v)>1:\n            Vs = nans_groups[k]\n            #cross_features=list(combinations(Vs, 2))\n            #make_corr(Vs)\n            grps= group_columns_by_correlation(train_df[Vs], threshold=0.8)\n            use=reduce_group(grps)\n            uses=uses+use\n            #make_corr(use)\n    else:\n        uses=uses+v\n    print('####### NAN count =',k)\nprint(uses)\nprint(len(uses))\nuses=uses+list(train_df.select_dtypes(include='category').columns)\nprint(len(uses))\ntrain_df=train_df[uses]","metadata":{"_kg_hide-output":true,"papermill":{"duration":62.857956,"end_time":"2024-05-08T14:03:07.090610","exception":false,"start_time":"2024-05-08T14:02:04.232654","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:13:10.601908Z","iopub.execute_input":"2024-05-09T15:13:10.602670Z","iopub.status.idle":"2024-05-09T15:14:07.889889Z","shell.execute_reply.started":"2024-05-09T15:13:10.602631Z","shell.execute_reply":"2024-05-09T15:14:07.888856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test","metadata":{"papermill":{"duration":0.019284,"end_time":"2024-05-08T14:03:07.130267","exception":false,"start_time":"2024-05-08T14:03:07.110983","status":"completed"},"tags":[]}},{"cell_type":"code","source":"data_store = {\n    \"df_base\": read_file(TEST_DIR / \"test_base.parquet\"),\n    \"depth_0\": [\n        read_file(TEST_DIR / \"test_static_cb_0.parquet\"),\n        read_files(TEST_DIR / \"test_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        read_files(TEST_DIR / \"test_applprev_1_*.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_a_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_b_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_c_1.parquet\", 1),\n        read_files(TEST_DIR / \"test_credit_bureau_a_1_*.parquet\", 1),\n        read_file(TEST_DIR / \"test_credit_bureau_b_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_other_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_person_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_deposit_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        read_file(TEST_DIR / \"test_credit_bureau_b_2.parquet\", 2),\n        read_files(TEST_DIR / \"test_credit_bureau_a_2_*.parquet\", 2),\n    ]\n}","metadata":{"papermill":{"duration":0.403478,"end_time":"2024-05-08T14:03:07.553025","exception":false,"start_time":"2024-05-08T14:03:07.149547","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:07.891082Z","iopub.execute_input":"2024-05-09T15:14:07.891373Z","iopub.status.idle":"2024-05-09T15:14:08.064902Z","shell.execute_reply.started":"2024-05-09T15:14:07.891347Z","shell.execute_reply":"2024-05-09T15:14:08.063779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = feature_eng(**data_store)\nprint(\"test data shape:\\t\", train_df.shape)","metadata":{"papermill":{"duration":0.075564,"end_time":"2024-05-08T14:03:07.649508","exception":false,"start_time":"2024-05-08T14:03:07.573944","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:08.069440Z","iopub.execute_input":"2024-05-09T15:14:08.069819Z","iopub.status.idle":"2024-05-09T15:14:08.117101Z","shell.execute_reply.started":"2024-05-09T15:14:08.069793Z","shell.execute_reply":"2024-05-09T15:14:08.116168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del data_store\ngc.collect()\ntest_df = test_df.select([col for col in train_df.columns if col != \"target\"])\nprint(\"train data shape:\\t\", train_df.shape)\nprint(\"test data shape:\\t\", test_df.shape)\n\ntest_df, cat_cols = to_pandas(test_df, cat_cols)\ntest_df = reduce_mem_usage(test_df)\n\ngc.collect()","metadata":{"papermill":{"duration":0.378922,"end_time":"2024-05-08T14:03:08.049009","exception":false,"start_time":"2024-05-08T14:03:07.670087","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:08.118309Z","iopub.execute_input":"2024-05-09T15:14:08.118693Z","iopub.status.idle":"2024-05-09T15:14:08.483607Z","shell.execute_reply.started":"2024-05-09T15:14:08.118659Z","shell.execute_reply":"2024-05-09T15:14:08.482643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_Id = test_df[\"case_id\"]","metadata":{"papermill":{"duration":0.027678,"end_time":"2024-05-08T14:03:08.097012","exception":false,"start_time":"2024-05-08T14:03:08.069334","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:08.484776Z","iopub.execute_input":"2024-05-09T15:14:08.485071Z","iopub.status.idle":"2024-05-09T15:14:08.489516Z","shell.execute_reply.started":"2024-05-09T15:14:08.485046Z","shell.execute_reply":"2024-05-09T15:14:08.488490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Check Data","metadata":{"papermill":{"duration":0.019133,"end_time":"2024-05-08T14:03:08.136539","exception":false,"start_time":"2024-05-08T14:03:08.117406","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df.tail()","metadata":{"papermill":{"duration":0.434903,"end_time":"2024-05-08T14:03:08.591249","exception":false,"start_time":"2024-05-08T14:03:08.156346","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:08.490819Z","iopub.execute_input":"2024-05-09T15:14:08.491486Z","iopub.status.idle":"2024-05-09T15:14:08.804919Z","shell.execute_reply.started":"2024-05-09T15:14:08.491453Z","shell.execute_reply":"2024-05-09T15:14:08.803916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.duplicated().sum()","metadata":{"papermill":{"duration":12.358683,"end_time":"2024-05-08T14:03:20.974970","exception":false,"start_time":"2024-05-08T14:03:08.616287","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:08.806063Z","iopub.execute_input":"2024-05-09T15:14:08.806930Z","iopub.status.idle":"2024-05-09T15:14:20.728452Z","shell.execute_reply.started":"2024-05-09T15:14:08.806902Z","shell.execute_reply":"2024-05-09T15:14:20.727530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* There is no duplicated data.","metadata":{"papermill":{"duration":0.02337,"end_time":"2024-05-08T14:03:21.022530","exception":false,"start_time":"2024-05-08T14:03:20.999160","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Target Variable","metadata":{"papermill":{"duration":0.026207,"end_time":"2024-05-08T14:03:21.073388","exception":false,"start_time":"2024-05-08T14:03:21.047181","status":"completed"},"tags":[]}},{"cell_type":"code","source":"label_counts = train_df[\"target\"].value_counts()\nlabel_counts","metadata":{"papermill":{"duration":0.044146,"end_time":"2024-05-08T14:03:21.141670","exception":false,"start_time":"2024-05-08T14:03:21.097524","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:20.729841Z","iopub.execute_input":"2024-05-09T15:14:20.730445Z","iopub.status.idle":"2024-05-09T15:14:20.745345Z","shell.execute_reply.started":"2024-05-09T15:14:20.730410Z","shell.execute_reply":"2024-05-09T15:14:20.744386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(5, 5))\nplt.pie(label_counts, labels=label_counts.index, autopct='%1.1f%%', startangle=90)\nplt.show()","metadata":{"papermill":{"duration":0.173537,"end_time":"2024-05-08T14:03:21.339214","exception":false,"start_time":"2024-05-08T14:03:21.165677","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:20.746488Z","iopub.execute_input":"2024-05-09T15:14:20.746835Z","iopub.status.idle":"2024-05-09T15:14:20.874751Z","shell.execute_reply.started":"2024-05-09T15:14:20.746796Z","shell.execute_reply":"2024-05-09T15:14:20.873364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Unbalanced data","metadata":{"papermill":{"duration":0.04536,"end_time":"2024-05-08T14:03:21.444064","exception":false,"start_time":"2024-05-08T14:03:21.398704","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# 4. Preprocessing","metadata":{"papermill":{"duration":0.02435,"end_time":"2024-05-08T14:03:21.493671","exception":false,"start_time":"2024-05-08T14:03:21.469321","status":"completed"},"tags":[]}},{"cell_type":"code","source":"y = train_df[\"target\"]\nweeks = train_df[\"WEEK_NUM\"]","metadata":{"papermill":{"duration":0.031419,"end_time":"2024-05-08T14:03:21.549860","exception":false,"start_time":"2024-05-08T14:03:21.518441","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:20.877009Z","iopub.execute_input":"2024-05-09T15:14:20.877786Z","iopub.status.idle":"2024-05-09T15:14:20.883380Z","shell.execute_reply.started":"2024-05-09T15:14:20.877736Z","shell.execute_reply":"2024-05-09T15:14:20.882147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.drop(columns = [\"case_id\", \"target\", \"WEEK_NUM\"], inplace = True)\ntest_df.drop(columns = [\"case_id\", \"WEEK_NUM\"], inplace = True)","metadata":{"papermill":{"duration":1.265375,"end_time":"2024-05-08T14:03:22.839366","exception":false,"start_time":"2024-05-08T14:03:21.573991","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:20.885410Z","iopub.execute_input":"2024-05-09T15:14:20.886269Z","iopub.status.idle":"2024-05-09T15:14:22.074744Z","shell.execute_reply.started":"2024-05-09T15:14:20.886223Z","shell.execute_reply":"2024-05-09T15:14:22.073933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[cat_cols] = train_df[cat_cols].astype(str)\ntest_df[cat_cols] = test_df[cat_cols].astype(str)","metadata":{"papermill":{"duration":7.818907,"end_time":"2024-05-08T14:03:30.685523","exception":false,"start_time":"2024-05-08T14:03:22.866616","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:22.075931Z","iopub.execute_input":"2024-05-09T15:14:22.076225Z","iopub.status.idle":"2024-05-09T15:14:29.276563Z","shell.execute_reply.started":"2024-05-09T15:14:22.076199Z","shell.execute_reply":"2024-05-09T15:14:29.275449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Encode Categorical Features","metadata":{"papermill":{"duration":0.026498,"end_time":"2024-05-08T14:03:30.738124","exception":false,"start_time":"2024-05-08T14:03:30.711626","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nlabel_encoder = LabelEncoder()\n\nfor col in train_df.select_dtypes(include=['object']):\n    train_df[col] = label_encoder.fit_transform(train_df[col])\n    \nfor col in test_df.select_dtypes(include=['object']):\n    test_df[col] = label_encoder.fit_transform(test_df[col])","metadata":{"papermill":{"duration":34.735073,"end_time":"2024-05-08T14:04:05.498644","exception":false,"start_time":"2024-05-08T14:03:30.763571","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:29.277866Z","iopub.execute_input":"2024-05-09T15:14:29.278158Z","iopub.status.idle":"2024-05-09T15:14:59.848671Z","shell.execute_reply.started":"2024-05-09T15:14:29.278133Z","shell.execute_reply":"2024-05-09T15:14:59.847546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Modelling","metadata":{"papermill":{"duration":0.02621,"end_time":"2024-05-08T14:04:05.551444","exception":false,"start_time":"2024-05-08T14:04:05.525234","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, accuracy_score, classification_report, roc_auc_score\nfrom sklearn.model_selection import cross_val_score\n\nfrom sklearn.model_selection import TimeSeriesSplit, GroupKFold, StratifiedGroupKFold\nfrom sklearn.impute import KNNImputer\n\nimport lightgbm as lgb\nimport catboost as cb\n\nfrom optuna.samplers import TPESampler\nimport optuna","metadata":{"papermill":{"duration":3.812167,"end_time":"2024-05-08T14:04:09.390028","exception":false,"start_time":"2024-05-08T14:04:05.577861","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:14:59.850038Z","iopub.execute_input":"2024-05-09T15:14:59.850341Z","iopub.status.idle":"2024-05-09T15:15:01.692900Z","shell.execute_reply.started":"2024-05-09T15:14:59.850316Z","shell.execute_reply":"2024-05-09T15:15:01.692008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(train_df, y, test_size=0.2, random_state=42)\n\n# Print the shapes of the training and testing datasets\nprint(\"X_train shape:\", X_train.shape)\nprint(\"X_test shape:\", X_test.shape)\nprint(\"y_train shape:\", y_train.shape)\nprint(\"y_test shape:\", y_test.shape)","metadata":{"papermill":{"duration":5.161458,"end_time":"2024-05-08T14:04:14.577156","exception":false,"start_time":"2024-05-08T14:04:09.415698","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:15:01.694160Z","iopub.execute_input":"2024-05-09T15:15:01.694549Z","iopub.status.idle":"2024-05-09T15:15:07.032152Z","shell.execute_reply.started":"2024-05-09T15:15:01.694514Z","shell.execute_reply":"2024-05-09T15:15:07.031025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.1. Feature Importance","metadata":{"papermill":{"duration":0.024568,"end_time":"2024-05-08T14:04:14.627850","exception":false,"start_time":"2024-05-08T14:04:14.603282","status":"completed"},"tags":[]}},{"cell_type":"code","source":"catb_classifier = cb.CatBoostClassifier(task_type='GPU')\ncatb_classifier.fit(X_train, y_train)\n\nfeature_importances = catb_classifier.feature_importances_","metadata":{"_kg_hide-output":true,"papermill":{"duration":165.46591,"end_time":"2024-05-08T14:07:00.117853","exception":false,"start_time":"2024-05-08T14:04:14.651943","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:15:07.033550Z","iopub.execute_input":"2024-05-09T15:15:07.033824Z","iopub.status.idle":"2024-05-09T15:17:19.975019Z","shell.execute_reply.started":"2024-05-09T15:15:07.033801Z","shell.execute_reply":"2024-05-09T15:17:19.973931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_importance_df = pd.DataFrame({'Feature': train_df.columns, 'Importance': feature_importances})\nfeature_importance_df = feature_importance_df.sort_values(by='Importance', ascending=False)","metadata":{"papermill":{"duration":0.051305,"end_time":"2024-05-08T14:07:00.204042","exception":false,"start_time":"2024-05-08T14:07:00.152737","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:19.977418Z","iopub.execute_input":"2024-05-09T15:17:19.978209Z","iopub.status.idle":"2024-05-09T15:17:19.985614Z","shell.execute_reply.started":"2024-05-09T15:17:19.978170Z","shell.execute_reply":"2024-05-09T15:17:19.984561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20, 50))\nplt.barh(feature_importance_df['Feature'], feature_importance_df['Importance'])\nplt.xlabel('Importance')\nplt.title('Feature Importances')\nplt.gca().invert_yaxis()\nplt.show()","metadata":{"papermill":{"duration":4.609088,"end_time":"2024-05-08T14:07:04.848689","exception":false,"start_time":"2024-05-08T14:07:00.239601","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:19.987008Z","iopub.execute_input":"2024-05-09T15:17:19.987400Z","iopub.status.idle":"2024-05-09T15:17:23.947215Z","shell.execute_reply.started":"2024-05-09T15:17:19.987366Z","shell.execute_reply":"2024-05-09T15:17:23.946280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"zero_importance_features = feature_importance_df[feature_importance_df['Importance'] == 0]['Feature']\n\nprint(\"Features with 0 importance: \\n\")\nfor feature in zero_importance_features:\n    print(feature)","metadata":{"papermill":{"duration":0.083456,"end_time":"2024-05-08T14:07:04.996097","exception":false,"start_time":"2024-05-08T14:07:04.912641","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:23.948345Z","iopub.execute_input":"2024-05-09T15:17:23.948638Z","iopub.status.idle":"2024-05-09T15:17:23.955900Z","shell.execute_reply.started":"2024-05-09T15:17:23.948612Z","shell.execute_reply":"2024-05-09T15:17:23.954863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* I achieved the best result by dropping these features.","metadata":{}},{"cell_type":"code","source":"feat_drop = ['deferredmnthsnum_166L',\n            'max_collater_typofvalofguarant_407M',\n            'max_inittransactioncode_279L',\n            'max_outstandingamount_354A',\n            'max_overdueamountmaxdateyear_994T',\n            'max_byoccupationinc_3656910L',\n            'max_empladdr_zipcode_114M',\n            'max_subjectrole_182M',\n            'totinstallast1m_4525188A',\n            'max_openingdate_313D',\n            'max_role_1084L',\n            'max_residualamount_488A',\n            'max_credacc_minhisbal_90A',\n            'assignmentdate_4527235D',\n            'numactivecreds_622L',\n            'pmtcount_693L',\n            'clientscnt_1071L']","metadata":{"execution":{"iopub.status.busy":"2024-05-09T15:17:23.957143Z","iopub.execute_input":"2024-05-09T15:17:23.957508Z","iopub.status.idle":"2024-05-09T15:17:23.966771Z","shell.execute_reply.started":"2024-05-09T15:17:23.957479Z","shell.execute_reply":"2024-05-09T15:17:23.965836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = train_df.drop(columns=feat_drop)\ntest_df = test_df.drop(columns=feat_drop)\n\nprint(\"Filtered train shape:\", train_df.shape)\nprint(\"Filtered test shape:\", test_df.shape)","metadata":{"papermill":{"duration":2.277688,"end_time":"2024-05-08T14:07:07.338781","exception":false,"start_time":"2024-05-08T14:07:05.061093","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:23.967896Z","iopub.execute_input":"2024-05-09T15:17:23.968203Z","iopub.status.idle":"2024-05-09T15:17:25.981656Z","shell.execute_reply.started":"2024-05-09T15:17:23.968169Z","shell.execute_reply":"2024-05-09T15:17:25.980608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(train_df, y, test_size=0.2, random_state=42)\n\n# Print the shapes of the training and testing datasets\nprint(\"X_train shape:\", X_train.shape)\nprint(\"X_test shape:\", X_test.shape)\nprint(\"y_train shape:\", y_train.shape)\nprint(\"y_test shape:\", y_test.shape)","metadata":{"papermill":{"duration":4.023929,"end_time":"2024-05-08T14:07:11.424352","exception":false,"start_time":"2024-05-08T14:07:07.400423","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:19:28.249486Z","iopub.execute_input":"2024-05-09T15:19:28.249924Z","iopub.status.idle":"2024-05-09T15:19:32.239265Z","shell.execute_reply.started":"2024-05-09T15:19:28.249892Z","shell.execute_reply":"2024-05-09T15:19:32.238286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2. Optuna Hyperparameter Tunning For LGBM","metadata":{"papermill":{"duration":0.058217,"end_time":"2024-05-08T14:07:11.544872","exception":false,"start_time":"2024-05-08T14:07:11.486655","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#def objective(trial):\n#    X_train, X_test, y_train, y_test = train_test_split(train_df, y, test_size=0.2, random_state=42)\n#\n#    params = {\n#        'boosting_type': 'gbdt',\n#        'objective': 'binary', \n#        'metric': 'binary_logloss', \n#        'verbosity': -1,\n#        'random_state': 42,\n#        'force_col_wise': True, \n#        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.1),\n#        'n_estimators': trial.suggest_int('n_estimators', 100, 1000),\n#        'max_depth': trial.suggest_int('max_depth', 6, 14),\n#        'num_leaves': trial.suggest_int('num_leaves', 20, 50),\n#        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0),\n#        'subsample': trial.suggest_float('subsample', 0.6, 1.0),\n#        'min_child_samples': trial.suggest_int('min_child_samples', 10, 50),\n#    }\n#\n#    model = lgb.LGBMClassifier(**params)\n#    model.fit(X_train, y_train)\n#\n#    y_pred = model.predict(X_test)\n#    accuracy = accuracy_score(y_test, y_pred)\n#    \n#    return accuracy\n#\n#study = optuna.create_study(direction=\"maximize\")\n#study.optimize(objective, n_trials=50)\n#\n#print(\"Best trial:\")\n#trial = study.best_trial\n#print(\"  Value: {}\".format(trial.value))\n#print(\"  Params: \")\n#for key, value in trial.params.items():\n#    print(\"    {}: {}\".format(key, value))","metadata":{"papermill":{"duration":0.067145,"end_time":"2024-05-08T14:07:11.672626","exception":false,"start_time":"2024-05-08T14:07:11.605481","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:25.994144Z","iopub.status.idle":"2024-05-09T15:17:25.994472Z","shell.execute_reply.started":"2024-05-09T15:17:25.994314Z","shell.execute_reply":"2024-05-09T15:17:25.994328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbm_params = {'learning_rate': 0.024535836002494142, \n               'n_estimators': 532, \n               'max_depth': 6, \n               'num_leaves': 28, \n               'colsample_bytree': 0.9141339012177984, \n               'subsample': 0.8092947399016269, \n               'min_child_samples': 41,\n               'device': 'gpu'}","metadata":{"papermill":{"duration":0.071313,"end_time":"2024-05-08T14:07:11.814801","exception":false,"start_time":"2024-05-08T14:07:11.743488","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:19:36.799387Z","iopub.execute_input":"2024-05-09T15:19:36.800047Z","iopub.status.idle":"2024-05-09T15:19:36.805145Z","shell.execute_reply.started":"2024-05-09T15:19:36.800012Z","shell.execute_reply":"2024-05-09T15:19:36.804096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3. Cross Validation","metadata":{"papermill":{"duration":0.059663,"end_time":"2024-05-08T14:07:11.934629","exception":false,"start_time":"2024-05-08T14:07:11.874966","status":"completed"},"tags":[]}},{"cell_type":"code","source":"n_splits = 5\ncv = StratifiedGroupKFold(n_splits=n_splits, shuffle=False)","metadata":{"papermill":{"duration":0.067152,"end_time":"2024-05-08T14:07:12.061329","exception":false,"start_time":"2024-05-08T14:07:11.994177","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:19:40.738189Z","iopub.execute_input":"2024-05-09T15:19:40.738544Z","iopub.status.idle":"2024-05-09T15:19:40.742985Z","shell.execute_reply.started":"2024-05-09T15:19:40.738517Z","shell.execute_reply":"2024-05-09T15:19:40.741897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fitted_models = {'catboost': [], 'lgbm': []}\ncv_scores = {'catboost': [], 'lgbm': []}\n\n# Iterate over each fold of the cross-validation\nfor idx_train, idx_valid in cv.split(train_df, y, groups=weeks):\n    X_train, y_train = train_df.iloc[idx_train], y.iloc[idx_train]\n    X_valid, y_valid = train_df.iloc[idx_valid], y.iloc[idx_valid]\n\n    print(\"Valid week range:\", (weeks.iloc[idx_valid].min(), weeks.iloc[idx_valid].max()))\n\n    # Train CatBoost model\n    catboost_model = cb.CatBoostClassifier(task_type = 'GPU')\n    catboost_model.fit(X_train, y_train, eval_set=(X_valid, y_valid), verbose=50, early_stopping_rounds=50)\n    fitted_models['catboost'].append(catboost_model)\n\n    # Evaluate CatBoost model\n    y_pred_valid_catboost = catboost_model.predict_proba(X_valid)[:, 1]\n    auc_score_catboost = roc_auc_score(y_valid, y_pred_valid_catboost)\n    cv_scores['catboost'].append(auc_score_catboost)\n\n    # Train LightGBM model\n    lgbm_model = lgb.LGBMClassifier(**lgbm_params)\n    lgbm_model.fit(X_train, y_train, eval_set=(X_valid, y_valid))\n    fitted_models['lgbm'].append(lgbm_model)\n\n    # Evaluate LightGBM model\n    y_pred_valid_lgbm = lgbm_model.predict_proba(X_valid)[:, 1]\n    auc_score_lgbm = roc_auc_score(y_valid, y_pred_valid_lgbm)\n    cv_scores['lgbm'].append(auc_score_lgbm)","metadata":{"_kg_hide-output":true,"papermill":{"duration":1715.503755,"end_time":"2024-05-08T14:35:47.626334","exception":false,"start_time":"2024-05-08T14:07:12.122579","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:19:44.092645Z","iopub.execute_input":"2024-05-09T15:19:44.093039Z","iopub.status.idle":"2024-05-09T15:45:54.572176Z","shell.execute_reply.started":"2024-05-09T15:19:44.093006Z","shell.execute_reply":"2024-05-09T15:45:54.571064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate and print average CV AUC scores for CatBoost and LightGBM\nprint(\"CV AUC scores (CatBoost):\", cv_scores['catboost'])\nprint(\"Average CV AUC score (CatBoost):\", np.mean(cv_scores['catboost']))\n\nprint(\"CV AUC scores (LightGBM):\", cv_scores['lgbm'])\nprint(\"Average CV AUC score (LightGBM):\", np.mean(cv_scores['lgbm']))","metadata":{"papermill":{"duration":0.085437,"end_time":"2024-05-08T14:35:47.786512","exception":false,"start_time":"2024-05-08T14:35:47.701075","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:46:13.721156Z","iopub.execute_input":"2024-05-09T15:46:13.721543Z","iopub.status.idle":"2024-05-09T15:46:13.727722Z","shell.execute_reply.started":"2024-05-09T15:46:13.721512Z","shell.execute_reply":"2024-05-09T15:46:13.726787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.4. Average Ensemble Model","metadata":{"papermill":{"duration":0.076674,"end_time":"2024-05-08T14:35:47.939579","exception":false,"start_time":"2024-05-08T14:35:47.862905","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class AveragingEnsembleModel:\n    def __init__(self, fitted_models):\n        self.fitted_models = fitted_models\n\n    def predict_proba(self, X):\n        predictions = np.zeros((X.shape[0], 2))\n        for model_type, models in self.fitted_models.items():\n            for model in models:\n                predictions += model.predict_proba(X)\n        return predictions / (len(self.fitted_models) * len(models))\n","metadata":{"papermill":{"duration":0.088057,"end_time":"2024-05-08T14:35:48.104798","exception":false,"start_time":"2024-05-08T14:35:48.016741","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:46:17.150492Z","iopub.execute_input":"2024-05-09T15:46:17.151171Z","iopub.status.idle":"2024-05-09T15:46:17.157580Z","shell.execute_reply.started":"2024-05-09T15:46:17.151136Z","shell.execute_reply":"2024-05-09T15:46:17.156538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ensemble_model = AveragingEnsembleModel(fitted_models)\ny_pred_test = ensemble_model.predict_proba(X_test)","metadata":{"papermill":{"duration":198.004192,"end_time":"2024-05-08T14:39:06.185461","exception":false,"start_time":"2024-05-08T14:35:48.181269","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:46:20.994002Z","iopub.execute_input":"2024-05-09T15:46:20.994775Z","iopub.status.idle":"2024-05-09T15:49:14.776321Z","shell.execute_reply.started":"2024-05-09T15:46:20.994740Z","shell.execute_reply":"2024-05-09T15:49:14.775412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ensemble_auc_score = roc_auc_score(y_test, y_pred_test[:, 1])\nprint(\"Ensemble Model AUC score:\", ensemble_auc_score)","metadata":{"papermill":{"duration":0.212231,"end_time":"2024-05-08T14:39:06.473803","exception":false,"start_time":"2024-05-08T14:39:06.261572","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:51:08.368370Z","iopub.execute_input":"2024-05-09T15:51:08.369091Z","iopub.status.idle":"2024-05-09T15:51:08.486613Z","shell.execute_reply.started":"2024-05-09T15:51:08.369054Z","shell.execute_reply":"2024-05-09T15:51:08.485623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.5. Weighted Voting Model","metadata":{"papermill":{"duration":0.075222,"end_time":"2024-05-08T14:39:06.628801","exception":false,"start_time":"2024-05-08T14:39:06.553579","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, ClassifierMixin\n\nclass WeightedVotingModel(BaseEstimator, ClassifierMixin):\n    def __init__(self, estimators, weights=None):\n        self.estimators = estimators\n        if weights is None:\n            self.weights = [1.0] * len(estimators)  # Equal weights by default\n        else:\n            self.weights = weights\n\n    def fit(self, X, y=None):\n        for estimator in self.estimators:\n            estimator.fit(X, y)\n        return self\n\n    def predict(self, X):\n        weighted_sum = np.zeros((X.shape[0], len(np.unique(y))))\n        for estimator, weight in zip(self.estimators, self.weights):\n            weighted_sum += estimator.predict_proba(X) * weight\n        return np.argmax(weighted_sum, axis=1)\n\n    def predict_proba(self, X):\n        weighted_sum = np.zeros((X.shape[0], len(np.unique(y))))\n        for estimator, weight in zip(self.estimators, self.weights):\n            weighted_sum += estimator.predict_proba(X) * weight\n        return weighted_sum / sum(self.weights)\n","metadata":{"papermill":{"duration":0.091589,"end_time":"2024-05-08T14:39:06.796712","exception":false,"start_time":"2024-05-08T14:39:06.705123","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:26.011415Z","iopub.status.idle":"2024-05-09T15:17:26.011795Z","shell.execute_reply.started":"2024-05-09T15:17:26.011616Z","shell.execute_reply":"2024-05-09T15:17:26.011632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fitted_catboost_models = fitted_models['catboost']\nfitted_lgbm_models = fitted_models['lgbm']\n\nvoting_model = WeightedVotingModel(estimators=fitted_catboost_models + fitted_lgbm_models)\n\n# y_pred_test = voting_model.predict_proba(X_test)\n\n# ensemble_auc_score = roc_auc_score(y_test, y_pred_test[:, 1])\n# print(\"Ensemble Model AUC score:\", ensemble_auc_score)","metadata":{"papermill":{"duration":0.088026,"end_time":"2024-05-08T14:39:06.963064","exception":false,"start_time":"2024-05-08T14:39:06.875038","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:26.013259Z","iopub.status.idle":"2024-05-09T15:17:26.013611Z","shell.execute_reply.started":"2024-05-09T15:17:26.013429Z","shell.execute_reply":"2024-05-09T15:17:26.013442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.6. Stacking","metadata":{"papermill":{"duration":0.077672,"end_time":"2024-05-08T14:39:07.118732","exception":false,"start_time":"2024-05-08T14:39:07.041060","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.ensemble import StackingClassifier\nfrom sklearn.linear_model import LogisticRegression","metadata":{"papermill":{"duration":0.175839,"end_time":"2024-05-08T14:39:07.373806","exception":false,"start_time":"2024-05-08T14:39:07.197967","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:26.014941Z","iopub.status.idle":"2024-05-09T15:17:26.015254Z","shell.execute_reply.started":"2024-05-09T15:17:26.015098Z","shell.execute_reply":"2024-05-09T15:17:26.015111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models = [('catboost', fitted_models['catboost'][0]), ('lgbm', fitted_models['lgbm'][0])]\nstacking_ensemble = StackingClassifier(estimators=models, final_estimator=LogisticRegression())\n\n# stacking_ensemble.fit(X_valid, y_valid)\n# y_pred_valid_stacking = stacking_ensemble.predict_proba(X_valid)[:, 1]\n\n\n# auc_score_stacking = roc_auc_score(y_valid, y_pred_valid_stacking)\n# print(\"AUC score (Stacking Ensemble):\", auc_score_stacking)","metadata":{"papermill":{"duration":458.001148,"end_time":"2024-05-08T14:46:45.450462","exception":false,"start_time":"2024-05-08T14:39:07.449314","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:17:26.016648Z","iopub.status.idle":"2024-05-09T15:17:26.016966Z","shell.execute_reply.started":"2024-05-09T15:17:26.016812Z","shell.execute_reply":"2024-05-09T15:17:26.016825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Submission\n---\n\nSince Average is the best of these three ensemble models, I submitted it.","metadata":{"papermill":{"duration":0.117565,"end_time":"2024-05-08T14:46:45.740839","exception":false,"start_time":"2024-05-08T14:46:45.623274","status":"completed"},"tags":[]}},{"cell_type":"code","source":"predictions = ensemble_model.predict_proba(test_df)[:, 1]","metadata":{"papermill":{"duration":0.236863,"end_time":"2024-05-08T14:46:46.095578","exception":false,"start_time":"2024-05-08T14:46:45.858715","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:51:29.158092Z","iopub.execute_input":"2024-05-09T15:51:29.158851Z","iopub.status.idle":"2024-05-09T15:51:29.250637Z","shell.execute_reply.started":"2024-05-09T15:51:29.158815Z","shell.execute_reply":"2024-05-09T15:51:29.249678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n    'case_id': test_Id,\n    'score': predictions\n})","metadata":{"papermill":{"duration":0.126874,"end_time":"2024-05-08T14:46:46.342395","exception":false,"start_time":"2024-05-08T14:46:46.215521","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:51:31.176842Z","iopub.execute_input":"2024-05-09T15:51:31.177941Z","iopub.status.idle":"2024-05-09T15:51:31.183121Z","shell.execute_reply.started":"2024-05-09T15:51:31.177896Z","shell.execute_reply":"2024-05-09T15:51:31.182063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)\n\nsubmission.head()","metadata":{"papermill":{"duration":0.132652,"end_time":"2024-05-08T14:46:46.590359","exception":false,"start_time":"2024-05-08T14:46:46.457707","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-05-09T15:51:33.591974Z","iopub.execute_input":"2024-05-09T15:51:33.592387Z","iopub.status.idle":"2024-05-09T15:51:33.608266Z","shell.execute_reply.started":"2024-05-09T15:51:33.592355Z","shell.execute_reply":"2024-05-09T15:51:33.607207Z"},"trusted":true},"execution_count":null,"outputs":[]}]}