{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30698,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"colab":{"provenance":[]}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G14 Notebook\n\nWelcome to the G14's notebook for the Home Credit Kaggle competition. The goal of this competition is to determine how likely a customer is going to default on an issued loan. The main difference between the [first](https://www.kaggle.com/c/home-credit-default-risk) and this competition is that now your submission will be scored with a custom metric that will take into account how well the model performs in future. A decline in performance will be penalized. The goal is to create a model that is stable and performs well in the future.\n\nIn this notebook you will see how to:\n* Load the data\n* Join tables with Polars - a DataFrame library implemented in Rust language, designed to be blazingy fast and memory efficient.  \n* Create simple aggregation features\n* Train a LightGBM model\n* Create a submission table\n\n## Load the data","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","id":"UT9kpBithazL"}},{"cell_type":"code","source":"import polars as pl\nimport numpy as np\nimport pandas as pd\nimport lightgbm as lgb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\n\ndataPath = \"/kaggle/input/home-credit-credit-risk-model-stability/\"","metadata":{"id":"Mm_NGuihhazM","execution":{"iopub.status.busy":"2024-04-22T01:29:26.790334Z","iopub.execute_input":"2024-04-22T01:29:26.790826Z","iopub.status.idle":"2024-04-22T01:29:26.797266Z","shell.execute_reply.started":"2024-04-22T01:29:26.790788Z","shell.execute_reply":"2024-04-22T01:29:26.79586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_table_dtypes(df: pl.DataFrame) -> pl.DataFrame:\n    # implement here all desired dtypes for tables\n    # the following is just an example\n    for col in df.columns:\n        # last letter of column name will help you determine the type\n        if col[-1] in (\"P\", \"A\"):\n            df = df.with_columns(pl.col(col).cast(pl.Float64).alias(col))\n\n    return df\n\ndef convert_strings(df: pd.DataFrame) -> pd.DataFrame:\n    for col in df.columns:\n        if df[col].dtype.name in ['object', 'string']:\n            df[col] = df[col].astype(\"string\").astype('category')\n            current_categories = df[col].cat.categories\n            new_categories = current_categories.to_list() + [\"Unknown\"]\n            new_dtype = pd.CategoricalDtype(categories=new_categories, ordered=True)\n            df[col] = df[col].astype(new_dtype)\n    return df","metadata":{"id":"So9JN4arhazM","execution":{"iopub.status.busy":"2024-04-22T01:29:26.799369Z","iopub.execute_input":"2024-04-22T01:29:26.799821Z","iopub.status.idle":"2024-04-22T01:29:26.809955Z","shell.execute_reply.started":"2024-04-22T01:29:26.79979Z","shell.execute_reply":"2024-04-22T01:29:26.808744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_basetable = pl.read_csv(dataPath + \"csv_files/train/train_base.csv\")\ntrain_static = pl.concat(\n    [\n        pl.read_csv(dataPath + \"csv_files/train/train_static_0_0.csv\").pipe(set_table_dtypes),\n        pl.read_csv(dataPath + \"csv_files/train/train_static_0_1.csv\").pipe(set_table_dtypes),\n    ],\n    how=\"vertical_relaxed\",\n)\ntrain_static_cb = pl.read_csv(dataPath + \"csv_files/train/train_static_cb_0.csv\").pipe(set_table_dtypes)\ntrain_person_1 = pl.read_csv(dataPath + \"csv_files/train/train_person_1.csv\").pipe(set_table_dtypes)\ntrain_credit_bureau_b_2 = pl.read_csv(dataPath + \"csv_files/train/train_credit_bureau_b_2.csv\").pipe(set_table_dtypes)","metadata":{"id":"BokoNCmAhazM","execution":{"iopub.status.busy":"2024-04-22T01:29:26.811373Z","iopub.execute_input":"2024-04-22T01:29:26.811792Z","iopub.status.idle":"2024-04-22T01:29:39.990671Z","shell.execute_reply.started":"2024-04-22T01:29:26.811729Z","shell.execute_reply":"2024-04-22T01:29:39.989609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_basetable = pl.read_csv(dataPath + \"csv_files/test/test_base.csv\")\ntest_static = pl.concat(\n    [\n        pl.read_csv(dataPath + \"csv_files/test/test_static_0_0.csv\").pipe(set_table_dtypes),\n        pl.read_csv(dataPath + \"csv_files/test/test_static_0_1.csv\").pipe(set_table_dtypes),\n        pl.read_csv(dataPath + \"csv_files/test/test_static_0_2.csv\").pipe(set_table_dtypes),\n    ],\n    how=\"vertical_relaxed\",\n)\ntest_static_cb = pl.read_csv(dataPath + \"csv_files/test/test_static_cb_0.csv\").pipe(set_table_dtypes)\ntest_person_1 = pl.read_csv(dataPath + \"csv_files/test/test_person_1.csv\").pipe(set_table_dtypes)\ntest_credit_bureau_b_2 = pl.read_csv(dataPath + \"csv_files/test/test_credit_bureau_b_2.csv\").pipe(set_table_dtypes)","metadata":{"id":"-W_f8daUhazN","execution":{"iopub.status.busy":"2024-04-22T01:29:39.993029Z","iopub.execute_input":"2024-04-22T01:29:39.993385Z","iopub.status.idle":"2024-04-22T01:29:40.035613Z","shell.execute_reply.started":"2024-04-22T01:29:39.993357Z","shell.execute_reply":"2024-04-22T01:29:40.034485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature engineering\n\nIn this part, we can see a simple example of joining tables via `case_id`. Here the loading and joining is done with polars library. Polars library is blazingly fast and has much smaller memory footprint than pandas.","metadata":{"id":"qwO6_7-ZhazN"}},{"cell_type":"code","source":"# We need to use aggregation functions in tables with depth > 1, so tables that contain num_group1 column or\n# also num_group2 column.\ntrain_person_1_feats_1 = train_person_1.group_by(\"case_id\").agg(\n    pl.col(\"mainoccupationinc_384A\").max().alias(\"mainoccupationinc_384A_max\"),\n    (pl.col(\"incometype_1044T\") == \"SELFEMPLOYED\").max().alias(\"mainoccupationinc_384A_any_selfemployed\")\n)\n\n# Here num_group1=0 has special meaning, it is the person who applied for the loan.\ntrain_person_1_feats_2 = train_person_1.select([\"case_id\", \"num_group1\", \"housetype_905L\"]).filter(\n    pl.col(\"num_group1\") == 0\n).drop(\"num_group1\").rename({\"housetype_905L\": \"person_housetype\"})\n\n# Here we have num_goup1 and num_group2, so we need to aggregate again.\ntrain_credit_bureau_b_2_feats = train_credit_bureau_b_2.group_by(\"case_id\").agg(\n    pl.col(\"pmts_pmtsoverdue_635A\").max().alias(\"pmts_pmtsoverdue_635A_max\"),\n    (pl.col(\"pmts_dpdvalue_108P\") > 31).max().alias(\"pmts_dpdvalue_108P_over31\")\n)\n\n# We will process in this examples only A-type and M-type columns, so we need to select them.\nselected_static_cols = []\nfor col in train_static.columns:\n    if col[-1] in (\"A\", \"M\"):\n        selected_static_cols.append(col)\nprint(selected_static_cols)\n\nselected_static_cb_cols = []\nfor col in train_static_cb.columns:\n    if col[-1] in (\"A\", \"M\"):\n        selected_static_cb_cols.append(col)\nprint(selected_static_cb_cols)\n\n# Join all tables together.\ndata = train_basetable.join(\n    train_static.select([\"case_id\"]+selected_static_cols), how=\"left\", on=\"case_id\"\n).join(\n    train_static_cb.select([\"case_id\"]+selected_static_cb_cols), how=\"left\", on=\"case_id\"\n).join(\n    train_person_1_feats_1, how=\"left\", on=\"case_id\"\n).join(\n    train_person_1_feats_2, how=\"left\", on=\"case_id\"\n).join(\n    train_credit_bureau_b_2_feats, how=\"left\", on=\"case_id\"\n)","metadata":{"id":"GIqz49P_hazN","outputId":"e34be669-f60e-444b-91c3-a572e792dedf","execution":{"iopub.status.busy":"2024-04-22T01:29:40.037357Z","iopub.execute_input":"2024-04-22T01:29:40.038095Z","iopub.status.idle":"2024-04-22T01:29:42.179243Z","shell.execute_reply.started":"2024-04-22T01:29:40.038057Z","shell.execute_reply":"2024-04-22T01:29:42.178159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_person_1_feats_1 = test_person_1.group_by(\"case_id\").agg(\n    pl.col(\"mainoccupationinc_384A\").max().alias(\"mainoccupationinc_384A_max\"),\n    (pl.col(\"incometype_1044T\") == \"SELFEMPLOYED\").max().alias(\"mainoccupationinc_384A_any_selfemployed\")\n)\n\ntest_person_1_feats_2 = test_person_1.select([\"case_id\", \"num_group1\", \"housetype_905L\"]).filter(\n    pl.col(\"num_group1\") == 0\n).drop(\"num_group1\").rename({\"housetype_905L\": \"person_housetype\"})\n\ntest_credit_bureau_b_2_feats = test_credit_bureau_b_2.group_by(\"case_id\").agg(\n    pl.col(\"pmts_pmtsoverdue_635A\").max().alias(\"pmts_pmtsoverdue_635A_max\"),\n    (pl.col(\"pmts_dpdvalue_108P\") > 31).max().alias(\"pmts_dpdvalue_108P_over31\")\n)\n\ndata_submission = test_basetable.join(\n    test_static.select([\"case_id\"]+selected_static_cols), how=\"left\", on=\"case_id\"\n).join(\n    test_static_cb.select([\"case_id\"]+selected_static_cb_cols), how=\"left\", on=\"case_id\"\n).join(\n    test_person_1_feats_1, how=\"left\", on=\"case_id\"\n).join(\n    test_person_1_feats_2, how=\"left\", on=\"case_id\"\n).join(\n    test_credit_bureau_b_2_feats, how=\"left\", on=\"case_id\"\n)","metadata":{"id":"0Dut4oHchazN","execution":{"iopub.status.busy":"2024-04-22T01:29:42.180551Z","iopub.execute_input":"2024-04-22T01:29:42.180863Z","iopub.status.idle":"2024-04-22T01:29:42.19658Z","shell.execute_reply.started":"2024-04-22T01:29:42.180837Z","shell.execute_reply":"2024-04-22T01:29:42.195357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Feature Analysis**","metadata":{"id":"xadvPZhFhazN"}},{"cell_type":"markdown","source":"# EDA","metadata":{"id":"clv3igyThazN"}},{"cell_type":"code","source":"pd_df = data.to_pandas() # dask, koalas : some alternatives foe EDA\nprint('Polar Dataframe')\nprint(type(pd_df))\nprint('Overall data analytics')\nprint(pd_df.describe())\nprint('Data types')\nprint(pd_df.dtypes)\n","metadata":{"id":"7XW-T4JUhazN","outputId":"86016ca0-12f3-492b-ef56-d7fc5d85b3f0","execution":{"iopub.status.busy":"2024-04-22T01:29:42.198023Z","iopub.execute_input":"2024-04-22T01:29:42.198356Z","iopub.status.idle":"2024-04-22T01:29:46.345395Z","shell.execute_reply.started":"2024-04-22T01:29:42.198328Z","shell.execute_reply":"2024-04-22T01:29:46.344198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Correlation Analysis\n\ncorrelation matrix performed on Int and Float values within dataset. No features have a strong relationship with the target feature.","metadata":{"id":"g60hL44jhazO"}},{"cell_type":"code","source":"\n\n#Correlation Analysis\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nnumerical_columns = pd_df.select_dtypes(include=['float64', 'int64']).columns\n\n# Subset DataFrame with numerical columns\nnumerical_df = pd_df[numerical_columns]\n\n# Example 1: Correlation Analysis\ncorrelation_matrix = numerical_df.corr()\n\n# Set up the matplotlib figure\nplt.figure(figsize=(12, 10))\n\n# Draw the heatmap\nsns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt=\".2f\")\n\n# Rotate column labels\nplt.xticks(rotation=45)  # Rotate x-axis labels by 45 degrees\n\n# Set the title\nplt.title('Correlation Matrix of Numerical Columns')\n\n# Show the plot\nplt.show()\n\nthreshold = 0.8  # Adjust as needed\n\n# Extract pairs of columns with high correlation\nhigh_correlation_pairs = []\n\n# Loop through the correlation matrix\nfor i in range(len(correlation_matrix.columns)):\n    for j in range(i+1, len(correlation_matrix.columns)):\n        if abs(correlation_matrix.iloc[i, j]) >= threshold:\n            pair = (correlation_matrix.columns[i], correlation_matrix.columns[j], correlation_matrix.iloc[i, j])\n            high_correlation_pairs.append(pair)\n\n# Print high correlation pairs\nprint(\"Pairs with correlation coefficient >= \", threshold)\nfor pair in high_correlation_pairs:\n    print(pair)\n\ntarget_correlation = correlation_matrix['target'].drop('target')  # Drop target variable itself\ntarget_correlation_with_target = target_correlation.abs().sort_values(ascending=False)\nprint(\"\\nCorrelation with Target:\")\nprint(target_correlation_with_target)","metadata":{"id":"U5J-p5CzhazO","outputId":"d6997d83-8ed6-4c7b-c47f-eb6ebde04f6b","execution":{"iopub.status.busy":"2024-04-22T01:29:46.346708Z","iopub.execute_input":"2024-04-22T01:29:46.347021Z","iopub.status.idle":"2024-04-22T01:29:56.027468Z","shell.execute_reply.started":"2024-04-22T01:29:46.346993Z","shell.execute_reply":"2024-04-22T01:29:56.026446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Drop the highly correlated columns from each pair, which aren't related to the target value","metadata":{"id":"e6vDYX1ghazO"}},{"cell_type":"code","source":"# Pairs with correlation coefficient >= 0.8\ncorrelated_pairs = [\n    ('MONTH', 'WEEK_NUM'),\n    ('amtinstpaidbefduel24m_4187115A', 'maxdebt4_972A'),\n    ('annuity_780A', 'credamount_770A'),\n    ('avginstallast24m_3658937A', 'avgpmtlast12m_4525200A'),\n    ('avgoutstandbalancel6m_4187114A', 'currdebt_22A'),\n    ('avgoutstandbalancel6m_4187114A', 'maxoutstandbalancel12m_4187113A'),\n    ('avgoutstandbalancel6m_4187114A', 'sumoutstandtotal_3546847A'),\n    ('avgoutstandbalancel6m_4187114A', 'sumoutstandtotalest_4493215A'),\n    ('avgoutstandbalancel6m_4187114A', 'totaldebt_9A'),\n    ('credamount_770A', 'disbursedcredamount_1113A'),\n    ('currdebt_22A', 'maxoutstandbalancel12m_4187113A'),\n    ('currdebt_22A', 'sumoutstandtotal_3546847A'),\n    ('currdebt_22A', 'sumoutstandtotalest_4493215A'),\n    ('currdebt_22A', 'totaldebt_9A'),\n    ('disbursedcredamount_1113A', 'inittransactionamount_650A'),\n    ('inittransactionamount_650A', 'price_1097A'),\n    ('maxoutstandbalancel12m_4187113A', 'sumoutstandtotal_3546847A'),\n    ('maxoutstandbalancel12m_4187113A', 'sumoutstandtotalest_4493215A'),\n    ('maxoutstandbalancel12m_4187113A', 'totaldebt_9A'),\n    ('maxpmtlast3m_4525190A', 'totinstallast1m_4525188A'),\n    ('sumoutstandtotal_3546847A', 'sumoutstandtotalest_4493215A'),\n    ('sumoutstandtotal_3546847A', 'totaldebt_9A'),\n    ('sumoutstandtotalest_4493215A', 'totaldebt_9A'),\n    ('pmtaverage_3A', 'pmtaverage_4527227A'),\n    ('pmtaverage_4527227A', 'pmtaverage_4955615A')\n]\n\n# Correlation of each column with the target variable\ncorrelation_with_target = {\n    'pmtaverage_4527227A': 0.056869,\n    'pmtaverage_3A': 0.052846,\n    'pmtssum_45A': 0.045376,\n    'amtinstpaidbefduel24m_4187115A': 0.045047,\n    'pmtaverage_4955615A': 0.043715,\n    'avglnamtstart24m_4525187A': 0.029916,\n    'disbursedcredamount_1113A': 0.027877,\n    'sumoutstandtotalest_4493215A': 0.027050,\n    'price_1097A': 0.026889,\n    'credamount_770A': 0.026277,\n    'maxdebt4_972A': 0.025744,\n    'sumoutstandtotal_3546847A': 0.021917,\n    'totaldebt_9A': 0.021764,\n    'currdebt_22A': 0.021745,\n    'totalsettled_863A': 0.019596,\n    'avgoutstandbalancel6m_4187114A': 0.017543,\n    'currdebtcredtyperange_828A': 0.017442,\n    'annuity_780A': 0.013838,\n    'maxannuity_159A': 0.013249,\n    'inittransactionamount_650A': 0.013203,\n    'pmts_pmtsoverdue_635A_max': 0.012941,\n    'lastotherlnsexpense_631A': 0.012272,\n    'maininc_215A': 0.011394,\n    'lastrejectcredamount_222A': 0.011019,\n    'lastotherinc_902A': 0.010642,\n    'maxannuity_4075009A': 0.010332,\n    'lastapprcredamount_781A': 0.008283,\n    'maxoutstandbalancel12m_4187113A': 0.006408,\n    'annuitynextmonth_57A': 0.006378,\n    'mainoccupationinc_384A_max': 0.006057,\n    'avginstallast24m_3658937A': 0.005615,\n    'downpmt_116A': 0.005212,\n    'MONTH': 0.004875,\n    'case_id': 0.003834,\n    'maxlnamtstart6m_4525199A': 0.003773,\n    'WEEK_NUM': 0.002969,\n    'totinstallast1m_4525188A': 0.002263,\n    'avgpmtlast12m_4525200A': 0.001701,\n    'maxinstallast24m_3658928A': 0.001042,\n    'maxpmtlast3m_4525190A': 0.000932\n}\n\n# Dictionary to store the selected column from each pair\nselected_columns = {}\n\n# Iterate through correlated pairs\nfor pair in correlated_pairs:\n    column1, column2 = pair\n    # Choose the column with the lower absolute correlation with the target\n    if abs(correlation_with_target[column1]) < abs(correlation_with_target[column2]):\n        selected_columns[pair] = column1\n    else:\n        selected_columns[pair] = column2\n\n# Print the selected columns\nfor pair, column in selected_columns.items():\n#     print(f\"Column from pair {pair} with lower correlation to target: {column}\")\n      print(column)\n","metadata":{"id":"ezx0TIzKhazO","outputId":"2a51bfd7-125d-4c41-8d55-448fe707c4b8","execution":{"iopub.status.busy":"2024-04-22T01:29:56.028845Z","iopub.execute_input":"2024-04-22T01:29:56.029222Z","iopub.status.idle":"2024-04-22T01:29:56.049152Z","shell.execute_reply.started":"2024-04-22T01:29:56.029189Z","shell.execute_reply":"2024-04-22T01:29:56.047991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"columns_to_drop = []\n\nfor pair, column in selected_columns.items():\n    columns_to_drop.append(column)\n\n# Drop columns from the main DataFrame\npd_df.drop(columns=columns_to_drop, inplace=True)\npd_df","metadata":{"id":"e0FDG6A6hazO","outputId":"4f9b7755-7c0c-46af-fdb4-e16c75a20fee","execution":{"iopub.status.busy":"2024-04-22T01:29:56.053418Z","iopub.execute_input":"2024-04-22T01:29:56.053735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"drop the less correlated columns","metadata":{"id":"_n83ddlEhazO"}},{"cell_type":"code","source":"\n# threshold = 0.05\n\n# # Identify features that have a correlation less than the threshold\n# # And make sure they were not already dropped\n# features_to_drop_based_on_threshold = [\n#     column for column, correlation in correlation_with_target.items()\n#     if abs(correlation) < threshold and column not in columns_to_drop\n# ]\n\n\n# pd_df = pd_df.drop(columns=features_to_drop_based_on_threshold, errors='ignore')\n\n# print(pd_df.columns)","metadata":{"id":"uWKgbabshazO","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Handling Nan Vaues","metadata":{"id":"XHDUzxQrhazP"}},{"cell_type":"code","source":"pd_df.isnull().sum()","metadata":{"id":"xePbLjeqhazP","outputId":"c80cc22e-1e68-404c-8162-cda53b39bc9f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"null_counts = pd_df.isnull().sum()\ncolumns_to_drop = null_counts[null_counts < 5].index\n\n# Drop rows where any of the specified columns have null values\npd_df_filtered = pd_df.dropna(subset=columns_to_drop)","metadata":{"id":"ee5OYtw4hazP","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"null_counts = pd_df_filtered.isnull().sum()\ncolumns_with_null_values = null_counts[null_counts > 0].index","metadata":{"id":"-n3A66sihazP","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Iterate through columns with null values\nfor column in columns_with_null_values:\n    print(\"Column:\", column)\n    print(pd_df_filtered[column].value_counts())\n    print()","metadata":{"id":"ZomDTQ2-hazP","outputId":"bc72589e-4b47-4f1e-87ca-6e160b0ba815","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Some info about the columns that need to be imputated\n1. amtinstpaidbefduel24m_4187115A - Number of instalments paid before due date in the last 24 months. {mode}\n2. ------------ avginstallast24m_3658937A --------------------- {avg}\n2. avglnamtstart24m_4525187A - Average loan amount in the last 24 months. {avg}\n3. avgoutstandbalancel6m_4187114A - Average outstanding balance of applicant for the last 6 months. {avg}\n4. avgpmtlast12m_4525200A - Average of payments made by the client in the last 12 months. {avg}\n5. inittransactionamount_650A - Initial transaction amount of the credit application.\n6. lastapprcredamount_781A - Credit amount from the client's last application.\n7. lastotherinc_902A - Amount of other income reported by the client in their last application.\n8. lastotherlnsexpense_631A - Monthly expenses on other loans from the last application.\n9. lastrejectcredamount_222A - Credit amount on last rejected application.\n10. maininc_215A - Client's primary income amount.\n11. maxannuity_159A - Maximum annuity previously obtained by client.\n12. maxannuity_4075009A - Maximal annuity offered to the client in the current application.\n13. maxdebt4_972A - Maximal principal debt of the client in the history older than 4 months.\n14. maxinstallast24m_3658928A- Maximum instalment in the last 24 months\n15. maxlnamtstart6m_4525199A - Maximum loan amount started in the last 6 months.\n16. maxoutstandbalancel12m_4187113A - Maximum outstanding balance in the last 12 months.\n17. maxpmtlast3m_4525190A - Maximum payment made by the client in the last 3 months.\n18. price_1097A - Credit price.\n19. sumoutstandtotal_3546847A - Sum of total outstanding amount.\n20. sumoutstandtotalest_4493215A - Sum of total outstanding amount.\n21. totinstallast1m_4525188A - Total amount of monthly instalments paid in the previous month.\n22. description_5085714M - Categorization of clients by credit bureau.\n23. education_1103M - Level of education of the client provided by external source.\n24. education_88M - Education level of the client.\n25. maritalst_385M - Marital status of the client.\n26. maritalst_893M - Marital status of the client\n27. pmtaverage_3A - Average of tax deductions.\n28. pmtaverage_4527227A - Average of tax deductions.\n29. pmtaverage_4955615A - Average of tax deductions.\n30. pmtssum_45A - Sum of tax deductions for the client.\n31. person_housetype - ??\n32. pmts_pmtsoverdue_635A_max - [pmts_pmtsoverdue_635A : Active contract that has overdue payments (num_group1 - existing contract, num_group2 - payment)].\n33. pmts_dpdvalue_108P_over31 - [pmts_dpdvalue_108P : Value of past due payment for active contract (num_group1 - existing contract, num_group2 - payment)]\n\nimo avg wont work for - amtinstpaidbefduel24m_4187115A, lastotherinc_902A, lastotherlnsexpense_631A, maxoutstandbalancel12m_4187113A, sumoutstandtotal_3546847A, sumoutstandtotalest_4493215A, description_5085714M, education_1103M, education_88M, maritalst_893M, pmtssum_45A, person_housetype, pmts_pmtsoverdue_635A_max","metadata":{"id":"h6DGBPIWhazP"}},{"cell_type":"code","source":"# Array containing column names where avg won't be a good fit for imputation\nspecified_columns = [\n    'amtinstpaidbefduel24m_4187115A', 'lastotherinc_902A', 'lastotherlnsexpense_631A',\n    'maxoutstandbalancel12m_4187113A', 'sumoutstandtotal_3546847A', 'sumoutstandtotalest_4493215A',\n    'description_5085714M', 'education_1103M', 'education_88M', 'maritalst_893M',\n    'pmtssum_45A', 'person_housetype', 'pmts_pmtsoverdue_635A_max'\n]\n\n# Select only numeric columns, excluding specified columns\nnumeric_columns = pd_df_filtered.select_dtypes(include=[np.number]).columns\n# columns_for_mean_imputation = [col for col in numeric_columns if col not in specified_columns and col in columns_with_null_values]\ncolumns_for_mean_imputation = [col for col in numeric_columns if col in columns_with_null_values]\n#  FOR NOW, REMOVED THE  col not in specified_columns and (I.E. doing mean imputation for all numerical columns containing NaN values)\n\n# Calculate means excluding specified columns\nmeans = pd_df_filtered[columns_for_mean_imputation].mean()\n\n# Replace NaN values with the mean for each eligible column\nfor column in columns_for_mean_imputation:\n    pd_df_filtered.loc[pd_df_filtered[column].isna(), column] = means[column]\n\n\nprint(pd_df_filtered)","metadata":{"id":"J5vaIKU1hazP","outputId":"51e73c61-25a8-46cc-d4cc-0069a10c7c81","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered.isnull().sum()","metadata":{"id":"QUhflug-hazP","outputId":"856f93ed-4594-4c80-a229-f3421d6af2e0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### One Hot Encoding the categorical data types and then handling the NaNs for that column","metadata":{"id":"REikOLbshazP"}},{"cell_type":"code","source":"pd_df_filtered['description_5085714M'].value_counts()","metadata":{"id":"9mPi7sDGhazP","outputId":"a042633c-d1b1-4979-926f-6c2168c80455","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered['education_1103M'].value_counts()","metadata":{"id":"qjZtQT9ghazQ","outputId":"e07db1da-b1e8-4add-c282-94a57131ae3a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered['education_88M'].value_counts()","metadata":{"id":"dN2rehW8hazQ","outputId":"197bb123-fc73-4151-c990-91f5ee9cc2c6","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered['maritalst_385M'].value_counts()","metadata":{"id":"KyIEZ3AihazQ","outputId":"5708eca4-121c-428d-e909-d491e6cd1cf1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered['maritalst_893M'].value_counts()","metadata":{"id":"pWXwvwjbhazQ","outputId":"388c7445-06c3-4994-94aa-12bdc04921ba","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered['person_housetype'].value_counts()","metadata":{"id":"9cbZXdxPhazQ","outputId":"531236b9-f65d-4938-f032-233bb300711f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered['pmts_dpdvalue_108P_over31'].value_counts()","metadata":{"id":"H58e6SrQhazQ","outputId":"fb21d2ea-855c-460e-ef73-ceec65e546b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#columns_to_one_hot_encode = [\n#    'description_5085714M', 'education_1103M', 'education_88M',\n#    'maritalst_385M', 'maritalst_893M', 'person_housetype',\n#    'pmts_dpdvalue_108P_over31'\n#]\n\n# Perform one-hot encoding and treat NaN as a separate category\n#for column in columns_to_one_hot_encode:\n    # Use get_dummies to encode the data, treating NaN as its own category\n    #encoded_columns = pd.get_dummies(pd_df_filtered[column], prefix=column, dummy_na=True)\n    # Drop the original column as it is now encoded\n    #pd_df_filtered = pd_df_filtered.drop(column, axis=1)\n    # Join the encoded columns to the original DataFrame\n    #pd_df_filtered = pd_df_filtered.join(encoded_columns)\n","metadata":{"id":"5bEFDeAyhazQ","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df_filtered.isnull().sum()","metadata":{"id":"PTHveDVihazR","outputId":"2e2b8964-d21e-403b-a645-85bfe162733a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Visualization","metadata":{"id":"xa9Bc4GPhazR"}},{"cell_type":"markdown","source":"1. Disbursed Credit Amount (disbursedcredamount_1113A):\n* Financial Insight: This variable represents the amount of credit disbursed to a client. It's a direct measure of the loan size, which is crucial in any credit risk model.\n* Risk Analysis: Larger loans might carry higher risk, depending on the income stability and financial health of the borrower. Analyzing the distribution of loan amounts can help in assessing the overall risk profile the financial institution is assuming.\n* Distribution Check: A histogram allows you to see the spread and central tendency of disbursed amounts, which can indicate typical loan sizes, outliers, or anomalies in loan disbursement (such as a very high number of small or large loans).","metadata":{"id":"992d2BlrhazR"}},{"cell_type":"markdown","source":"2. Main Occupation Income (mainoccupationinc_384A_max):\n* Income Relationship: Income is a fundamental factor in determining a borrower's ability to repay a loan. Higher incomes generally correlate with lower default rates, but the distribution of income needs to be understood fully.\n* Affordability Analysis: By understanding income levels, the lender can better assess affordability ratios for loan repayments.\n* Economic Factors: Income distributions can also reflect broader economic conditions and employment stability within the borrower population.","metadata":{"id":"DWqXKrcIhazR"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n# Histogram for Disbursed Credit Amount\nplt.figure(figsize=(10, 6))\nplt.hist(pd_df_filtered['disbursedcredamount_1113A'].dropna(), bins=50, color='blue', alpha=0.7)\nplt.title('Histogram of Disbursed Credit Amounts')\nplt.xlabel('Disbursed Credit Amount')\nplt.ylabel('Frequency')\nplt.grid(True)\nplt.show()\n\n# Histogram for Main Occupation Income\nplt.figure(figsize=(10, 6))\nplt.hist(pd_df_filtered['mainoccupationinc_384A_max'].dropna(), bins=50, color='green', alpha=0.7)\nplt.title('Histogram of Main Occupation Income')\nplt.xlabel('Income')\nplt.ylabel('Frequency')\nplt.grid(True)\nplt.show()","metadata":{"id":"VfbC-iPrhazS","outputId":"7cb9ad60-fec7-466f-acb8-0fc5f510c187","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Analysis of the plots\n\n**Histogram of Disbursed Credit Amounts:**\n* Right-Skewed Distribution: The histogram shows a distribution that is heavily skewed to the right, meaning most of the disbursed credit amounts are on the lower end of the scale.\n* Common Loan Amounts: There is a high frequency of smaller loan amounts, suggesting that the majority of loans disbursed are of smaller value, which could be indicative of the institution's lending strategy or customer demographics.\n* Outliers: There are fewer loans disbursed with very high amounts, as visible by the long tail to the right. This suggests that while larger loans are part of the product mix, they are relatively rare.\n* Risk Diversification: The concentration of smaller loans might indicate a strategy to diversify risk by not concentrating the portfolio in high-value loans, which could represent a higher individual default risk.\n\n**Histogram of Main Occupation Income:**\n* Multi-Modal Distribution: The income histogram appears to be multi-modal, with several peaks, indicating there are groups within the population that have distinct income levels.\n* Income Groups: The different peaks might suggest the presence of various customer segments, possibly entry-level, mid-career, and high-earning individuals.\n* Wider Income Range: Unlike the disbursed credit amounts, income values spread across a wider range, though still right-skewed. This indicates variability in the economic status of the applicants.\n* Affordability Analysis: The institution can use these income segments for affordability analysis, by determining which income levels have higher or lower risk of default.\n\n**General Observations:**\n* Data Transformation for Modeling: The skewness in both histograms indicates that transformations (like logarithmic or square root) might be useful during the data preprocessing to normalize the distributions for better model performance.\n* Loan to Income Ratio: Combining these two distributions could lead to a derived feature of loan to income ratio, which might be a significant predictor in assessing the likelihood of a default.","metadata":{"id":"ndwnOyS4hazS"}},{"cell_type":"markdown","source":"# Handle Class imbalance and avoid possible overfitting, ONLY for target (y) column","metadata":{"id":"5YSlKikVhazS"}},{"cell_type":"code","source":"pd_df_filtered['target'].value_counts()","metadata":{"id":"iUK2GBAshazS","outputId":"eea3dd49-283f-4988-cb11-b820adc843ac","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### drop columns that have less correlation with target, i.e. below the threshold = 0.4","metadata":{"id":"9C8sEFsGhazT"}},{"cell_type":"code","source":"# Set display options to show all rows and columns without truncation\npd.set_option('display.max_rows', None)\npd.set_option('display.max_columns', None)\npd.set_option('display.width', None)\npd.set_option('display.max_colwidth', None)\n\nnum_columns = pd_df_filtered.shape[1]\n\n# Print the number of columns\nprint(\"Number of columns:\", num_columns)\npd_df_filtered.dtypes","metadata":{"id":"1J1DElcqhazT","outputId":"c0193922-5533-41e6-857f-58de2d163cab","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(pd_df_filtered.columns)","metadata":{"id":"jl9itagnPxQZ","outputId":"2dc9af8e-cb72-4330-b363-7735cd63195c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df = pd_df_filtered.drop('date_decision', axis=1)","metadata":{"id":"SCjSqr94hazT","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# categorical_columns = pd_df.select_dtypes(include=['object', 'bool']).columns\n\n# Perform one-hot encoding on these categorical columns\n# pd_df = pd.get_dummies(pd_df, columns=categorical_columns, drop_first=True)\n","metadata":{"id":"HClbZ6NUhazT","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pd_df","metadata":{"id":"njEWNlsyhazT","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(pd_df.columns)","metadata":{"id":"N1-GTyM_NhTe","outputId":"b8d61ce6-f820-4cb7-ff46-eab698b73f36","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df.isnull().sum()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_df.dtypes","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\ncols = pd_df.select_dtypes(include=['category', 'object']).columns\n\nfor col in cols:\n    le = LabelEncoder()\n    pd_df[col] = le.fit_transform(pd_df[col])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(pd_df.columns)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from imblearn.over_sampling import SMOTE\nfrom sklearn.model_selection import train_test_split\n\n\nX = pd_df.drop('target', axis=1)\ny = pd_df['target']\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)\nX_valid, X_test, y_valid, y_test = train_test_split(X_test, y_test, test_size=0.5, stratify=y_test, random_state=42)\n\n\nsmote = SMOTE(random_state=42)\n\n\nX_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)\nX_valid_smote, y_valid_smote = smote.fit_resample(X_valid, y_valid)\nX_test_smote, y_test_smote = smote.fit_resample(X_test, y_test)\n\n# Checking the new class distribution\nprint(pd.Series(y_train_smote).value_counts())\n","metadata":{"id":"7r71r31uhazT","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature importance with Random Forrest","metadata":{"id":"mE18hNJfhazT"}},{"cell_type":"code","source":"# from sklearn.model_selection import train_test_split\n# from sklearn.ensemble import RandomForestRegressor\n# from sklearn.metrics import mean_squared_error\n\n# numerical_columns = pd_df.select_dtypes(include=['float64', 'int64']).columns\n\n# print(numerical_df)\n# # Split the data into features (X) and target variable (y)\n# X = numerical_df.drop(columns=['target'])\n# y = numerical_df['target']\n\n\n# # Adjust test_size parameter to ensure enough data for training\n# test_size = 0.2 if len(X) > 5 else 0.1  # For example, set test_size to 0.1 if there are fewer than 5 samples\n\n# # Split the data into training and testing sets\n# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=42)\n\n# # Initialize and train the Random Forest model\n# model = RandomForestRegressor(n_estimators=100, random_state=42)\n# model.fit(X_train, y_train)\n\n# # Predict on the test set\n# y_pred = model.predict(X_test)\n\n# # Evaluate the model\n# mse = mean_squared_error(y_test, y_pred)\n# print(\"Mean Squared Error:\", mse)\n\n# # Feature importance\n# feature_importance = model.feature_importances_\n# feature_importance_df = pd.DataFrame({'Feature': X.columns, 'Importance': feature_importance})\n# feature_importance_df = feature_importance_df.sort_values(by='Importance', ascending=False)\n# print(\"Feature Importance:\")\n# print(feature_importance_df)","metadata":{"id":"BAxjExvvhazT","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# case_ids = data[\"case_id\"].unique().shuffle(seed=1)\n# case_ids_train, case_ids_test = train_test_split(case_ids, train_size=0.6, random_state=1)\n# case_ids_valid, case_ids_test = train_test_split(case_ids_test, train_size=0.5, random_state=1)\n\n# cols_pred = []\n# for col in data.columns:\n#     if col[-1].isupper() and col[:-1].islower():\n#         cols_pred.append(col)\n\n# def from_polars_to_pandas(case_ids: pl.DataFrame) -> pl.DataFrame:\n#     return (\n#         data.filter(pl.col(\"case_id\").is_in(case_ids))[[\"case_id\", \"WEEK_NUM\", \"target\"]].to_pandas(),\n#         data.filter(pl.col(\"case_id\").is_in(case_ids))[cols_pred].to_pandas(),\n#         data.filter(pl.col(\"case_id\").is_in(case_ids))[\"target\"].to_pandas()\n#     )\n# base_train, X_train, y_train = from_polars_to_pandas(case_ids_train)\n# base_valid, X_valid, y_valid = from_polars_to_pandas(case_ids_valid)\n# base_test, X_test, y_test = from_polars_to_pandas(case_ids_test)\n\n# for df in [X_train, X_valid, X_test]:\n#     df = convert_strings(df)","metadata":{"id":"duvAtn-qhazT","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_train","metadata":{"id":"We-_bpRxhazU","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# y_train.value_counts()","metadata":{"id":"3lQLGiIrhazU","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(f\"Train: {X_train.shape}\")\n# print(f\"Valid: {X_valid.shape}\")\n# print(f\"Test: {X_test.shape}\")","metadata":{"id":"-N4JRVBPhazU","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#**Training Models**\n\nIn this section, we are going to train several ensemble models and apply cross validation to select the best model based on performance.\n\n* LightGBM\n* XGboost\n* CatBoost\n* AdaBoost\n* HistGradientBoostingClassifier\n* SVM\n","metadata":{"id":"r20H2IcmxR_D"}},{"cell_type":"markdown","source":"## LightGBM","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\nlgb_train = lgb.Dataset(X_train_smote, label=y_train_smote)\nlgb_valid = lgb.Dataset(X_valid_smote, label=y_valid_smote, reference=lgb_train)\n\nparams = {\n    \"boosting_type\": \"gbdt\",  # Gradient boosting decision tree algorithm\n    \"objective\": \"binary\",  # Binary log loss classification (binary classification problem)\n    \"metric\": \"auc\",  # The metric to be used for validation data is AUC (Area Under Curve)\n    \"max_depth\": 6,  # Maximum tree depth for base learners\n    \"num_leaves\": 40,  # Maximum tree leaves for base learners\n    \"learning_rate\": 0.01,  # Boosting learning rate\n    \"feature_fraction\": 0.8,  # Subsample ratio of columns when constructing each tree\n    \"bagging_fraction\": 0.7,  # Subsample ratio of the training instance (prevents overfitting)\n    \"bagging_freq\": 5,  # Frequency for bagging (k means perform bagging at every k iteration)\n    \"n_estimators\": 2050,  # Number of boosted trees to fit\n    \"verbose\": -1,  # Controls the level of LightGBM's verbosity (less verbose)\n    \"min_child_samples\": 30,  # Minimum number of data needed in a child (leaf)\n    \"min_split_gain\": 0.1,  # Minimum loss reduction required to make a further partition on a leaf node of the tree\n    \"reg_alpha\": 0.51,  # L1 regularization term on weights (increases model's generalization)\n    \"reg_lambda\": 0.5  # L2 regularization term on weights (prevents overfitting)\n}\n\n\ngbm = lgb.train(\n    params,\n    lgb_train,\n    valid_sets=lgb_valid,\n    callbacks=[lgb.log_evaluation(50)]\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\n\n# Create a K-Fold object with 10 splits\ncv = KFold(n_splits=2, shuffle=True, random_state=42)\n\n# Define a list to store AUC values for each fold\nauc_scores_light = []\nfitted_models_lgb= []\nbest_model = None\nbest_auc = -1\nmodel_filename = \"best_lightgbm_model.joblib\"\n\n# Loop over each fold to train and evaluate the model\nfor fold_number, (train_idx, valid_idx) in enumerate(cv.split(X_train_smote)):\n    # Split into training and validation sets\n    X_train_fold = X_train_smote.iloc[train_idx]\n    y_train_fold = y_train_smote.iloc[train_idx]\n\n    X_valid_fold = X_train_smote.iloc[valid_idx]\n    y_valid_fold = y_train_smote.iloc[valid_idx]\n\n    # LightGBM datasets\n    lgb_train = lgb.Dataset(X_train_fold, label=y_train_fold)\n    lgb_valid = lgb.Dataset(X_valid_fold, label=y_valid_fold, reference=lgb_train)\n\n    # Train the LightGBM model\n    model = lgb.train(\n        params,\n        lgb_train,\n        valid_sets=[lgb_valid],  # Validation dataset\n        callbacks=[lgb.log_evaluation(50),lgb.early_stopping(20)]  # Log every 50 iterations\n    )\n    \n    fitted_models_lgb.append(model)\n\n    # Predict on the validation set\n    predictions = model.predict(X_valid_fold)\n\n    # Calculate AUC and store it\n    auc = roc_auc_score(y_valid_fold, predictions)  # Calculate AUC for this fold\n    auc_scores_light.append(auc)\n\n    print(f\"Fold {fold_number + 1} AUC: {auc}\")\n    print(\"Maximum CV AUC score: \", max(auc_scores_light))\n    \n    # Check if this fold's AUC is the best\n    if auc > best_auc:\n        best_auc = auc\n        best_model = model  # Store the best model\n    # Save the best model to a file\n    if best_model:\n        joblib.dump(best_model, model_filename)  # Save the best model\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Maximum CV AUC score:  0.9835367989135522\n# Plot the AUC scores for each fold=2\nn_splits=2\nplt.plot(range(1, 3), auc_scores_light, marker='o', linestyle='-', color='b')\nplt.title(\"AUC for each Cross-Validation Fold\")\nplt.xlabel(\"Fold\")\nplt.ylabel(\"AUC\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get feature importance from the best model\nfeature_importance = best_model.feature_importance(importance_type='split')  # 'split' or 'gain'\nfeature_names = X_train_smote.columns\n\n# Plot feature importance\nplt.figure(figsize=(10, 6))\nplt.barh(feature_names, feature_importance, color='skyblue')\nplt.xlabel(\"Feature Importance\")\nplt.ylabel(\"Feature\")\nplt.title(\"Feature Importance (LightGBM)\")\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nfrom sklearn.metrics import roc_auc_score\n\n# Load the best model from the saved file\nbest_model = joblib.load(\"best_lightgbm_model.joblib\")\n\n# Use the model to predict on the untrained data\npredictions_test = best_model.predict(X_test_smote)\n\n# Calculate AUC score on the test set\nauc_test = roc_auc_score(y_test_smote, predictions_test)\n\nprint(f\"Test AUC: {auc_test}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CatBoost","metadata":{}},{"cell_type":"code","source":"import catboost as cb\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\n\n# CatBoost parameters\nparams = {\n    'iterations': 3000,\n    'learning_rate': 0.03,\n    'depth': 6,\n    'loss_function': 'Logloss',\n    'eval_metric': 'AUC',\n    'verbose': 50\n}\n\n# K-Fold configuration\nn_splits = 2\ncv = KFold(n_splits=n_splits, shuffle=True, random_state=42)\n\n# List to store AUC scores for each fold\nauc_scores_catboost = []\nfitted_models_cat = []\nbest_model = None\nbest_auc = -1\nmodel_filename = \"best_catboost_model.joblib\"\n\n# Loop through each fold\nfor fold_number, (train_idx, valid_idx) in enumerate(cv.split(X_train_smote)):\n    # Training and validation sets\n    X_train_fold = X_train_smote.iloc[train_idx]\n    y_train_fold = y_train_smote.iloc[train_idx]\n\n    X_valid_fold = X_train_smote.iloc[valid_idx]\n    y_valid_fold = y_train_smote.iloc[valid_idx]\n\n    # Convert data to Pool objects\n    train_pool = cb.Pool(X_train_fold, label=y_train_fold)\n    valid_pool = cb.Pool(X_valid_fold, label=y_valid_fold)\n\n    # Initialize the CatBoost model\n    model = cb.CatBoostClassifier(**params)\n\n    # Fit the model with training and validation data\n    model.fit(\n        train_pool,\n        eval_set=valid_pool,\n        early_stopping_rounds=20  # Early stopping if no improvement\n    )\n    \n    fitted_models_cat.append(model)\n\n    # Predictions and AUC calculation for validation set\n    predictions = model.predict_proba(X_valid_fold)[:, 1]\n    auc = roc_auc_score(y_valid_fold, predictions)\n    auc_scores_catboost.append(auc)\n\n    print(f\"Fold {fold_number + 1} AUC: {auc}\")\n    print(\"Maximum CV AUC score: \", max(auc_scores_catboost))\n    \n    # Check if this fold's AUC is the best\n    if auc > best_auc:\n        best_auc = auc\n        best_model = model  # Store the best model\n    # Save the best model to a file\n    if best_model:\n        joblib.dump(best_model, model_filename)  # Save the best model\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the AUC scores for each fold\nplt.plot(range(1, 3), auc_scores_catboost, marker='o', linestyle='-', color='b')\nplt.title(\"AUC for each Cross-Validation Fold\")\nplt.xlabel(\"Fold\")\nplt.ylabel(\"AUC\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nfrom sklearn.metrics import roc_auc_score\n\n# Load the best model from the saved file\nbest_model = joblib.load(\"best_catboost_model.joblib\")\n\n# Use the model to predict on the untrained data\npredictions_test = best_model.predict(X_test_smote)\n\n# Calculate AUC score on the test set\nauc_test = roc_auc_score(y_test_smote, predictions_test)\n\nprint(f\"Test AUC: {auc_test}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## HistGradientBoostingClassifier","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import HistGradientBoostingClassifier\n\nn_splits = 2  # Number of folds\ncv = KFold(n_splits=n_splits, shuffle=True, random_state=42)  # Shuffling to randomize splits\n\n# List to store AUC scores\nauc_scores_hist = []\nfitted_models_hist = []\nbest_model_hist = None\nbest_auc_hist = -1\nmodel_filename = \"best_histboost_model.joblib\"\n\n# Loop through each fold\nfor fold_number, (train_idx, valid_idx) in enumerate(cv.split(X_train_smote)):\n    # Split into training and validation sets\n    X_train_fold = X_train_smote.iloc[train_idx]\n    y_train_fold = y_train_smote.iloc[train_idx]\n\n    X_valid_fold = X_train_smote.iloc[valid_idx]\n    y_valid_fold = y_train_smote.iloc[valid_idx]\n    \n    # Initialize the model\n    model = HistGradientBoostingClassifier(\n    max_iter=1000,  # Number of boosting iterations\n    learning_rate=0.01,\n    max_depth=10,\n    random_state=42,\n    verbose=0\n)\n\n    # Train the model\n    model.fit(X_train_fold, y_train_fold)\n    fitted_models_hist.append(model)\n\n    # Predict probabilities for the positive class\n    y_pred_proba = model.predict_proba(X_valid_fold)[:, 1]\n\n    # Calculate AUC\n    auc = roc_auc_score(y_valid_fold, y_pred_proba)\n    auc_scores_hist.append(auc)\n\n    print(f\"Fold {fold_number + 1} AUC: {auc}\")\n    print(\"Maximum CV AUC score: \", max(auc_scores_hist))\n    \n    # Check if this fold's AUC is the best\n    if auc > best_auc_hist:\n        best_auc_hist = auc\n        best_model_hist = model  # Store the best model\n    # Save the best model to a file\n    if best_model_hist:\n        try:\n            joblib.dump(best_model_hist, \"best_histboost_model.joblib\")\n        except Exception as e:\n            print(f\"Error during save: {e}\")\n        ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the AUC scores for each fold\nplt.plot(range(1, 3), auc_scores_hist, marker='o', linestyle='-', color='b')\nplt.title(\"AUC for each Cross-Validation Fold\")\nplt.xlabel(\"Fold\")\nplt.ylabel(\"AUC\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nfrom sklearn.metrics import roc_auc_score\n\n# Load the best model from the saved file\nbest_model = joblib.load(\"best_histboost_model.joblib\")\n\n# Use the model to predict on the untrained data\npredictions_test = best_model.predict(X_test_smote)\n\n# Calculate AUC score on the test set\nauc_test = roc_auc_score(y_test_smote, predictions_test)\n\nprint(f\"Test AUC: {auc_test}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## XGBoost","metadata":{}},{"cell_type":"code","source":"import joblib\nimport matplotlib.pyplot as plt\nimport xgboost as xgb\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\n\n# Create a K-Fold object with 3 splits\ncv = KFold(n_splits=3, shuffle=True, random_state=42)\n\n# Define a list to store AUC values for each fold\nauc_scores_xgboost = []\nfitted_models_xgb=[]\nbest_model = None\nbest_auc = -1\nmodel_filename = \"best_xgboost_model.joblib\"\n\n# Define parameters for XGBoost\nparams_xgb = {\n    \"booster\": \"gbtree\",\n    \"objective\": \"binary:logistic\",\n    \"eval_metric\": \"auc\",\n    \"max_depth\": 10,\n    \"learning_rate\": 0.05,\n    \"n_estimators\": 2050,\n    \"colsample_bytree\": 0.8,\n    \"colsample_bynode\": 0.8,\n    \"alpha\": 0.1,  \n    \"lambda\": 10,  \n    \"random_state\": 42,\n    \"verbosity\": 0,\n    \"enable_categorical\": True,\n}\n\n# Loop over each fold to train and evaluate the model\nfor fold_number, (train_idx, valid_idx) in enumerate(cv.split(X_train_smote)):\n    # Split into training and validation sets\n    X_train_fold = X_train_smote.iloc[train_idx]\n    y_train_fold = y_train_smote.iloc[train_idx]\n\n    X_valid_fold = X_train_smote.iloc[valid_idx]\n    y_valid_fold = y_train_smote.iloc[valid_idx]\n\n    # Create DMatrix for XGBoost\n    dtrain = xgb.DMatrix(X_train_fold, label=y_train_fold)\n    dvalid = xgb.DMatrix(X_valid_fold, label=y_valid_fold)\n\n    # Train the XGBoost model\n    bst = xgb.train(\n        params_xgb,\n        dtrain,\n        num_boost_round=1000,  # Maximum number of boosting rounds\n        evals=[(dvalid, 'validation')],  # Evaluation set\n        early_stopping_rounds=10,  # Early stopping\n        verbose_eval=10  # Print evaluation results every 50 rounds\n    )\n    \n    fitted_models_xgb.append(bst)\n\n    # Predict on the validation set\n    predictions = bst.predict(dvalid)\n\n    # Calculate AUC and store it\n    auc = roc_auc_score(y_valid_fold, predictions)  # Calculate AUC for this fold\n    auc_scores_xgboost.append(auc)\n\n    print(f\"Fold {fold_number + 1} AUC: {auc}\")\n    print(\"Maximum CV AUC score: \", max(auc_scores_xgboost))\n\n    # Check if this fold's AUC is the best\n    if auc > best_auc:\n        best_auc = auc\n        best_model = bst  # Store the best model\n\n# Save the best model to a file\nif best_model:\n    joblib.dump(best_model, model_filename)  # Save the best model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the AUC scores for each fold\nplt.plot(range(3), auc_scores_xgboost, marker='o', linestyle='-', color='b')\nplt.title(\"AUC for each Cross-Validation Fold\")\nplt.xlabel(\"Fold\")\nplt.ylabel(\"AUC\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dtest = xgb.DMatrix(X_test_smote, label=y_test_smote)\n\nimport joblib\nfrom sklearn.metrics import roc_auc_score\n\n# Load the best model from the saved file\nbest_model = joblib.load(\"best_xgboost_model.joblib\")\n\n# Use the model to predict on the untrained data\npredictions_test = best_model.predict(dtest)\n\n# Calculate AUC score on the test set\nauc_test = roc_auc_score(y_test_smote, predictions_test)\n\nprint(f\"Test AUC: {auc_test}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## AdaBoost","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import AdaBoostClassifier\n\n# Create a K-Fold object with 3 splits\ncv = KFold(n_splits=3, shuffle=True, random_state=42)\n\n# Define a list to store AUC values for each fold\nauc_scores_adaboost = []\nfitted_models_ada=[]\nbest_model = None\nbest_auc = -1\nmodel_filename = \"best_adaboost_model.joblib\"\n\n# Define parameters for AdaBoost\nparams_adaboost = {\n    'n_estimators': 2050,\n    'learning_rate': 0.05,\n    'random_state': 42,\n}\n\n# Loop over each fold to train and evaluate the model\nfor fold_number, (train_idx, valid_idx) in enumerate(cv.split(X_train_smote)):\n    # Split into training and validation sets\n    X_train_fold = X_train_smote.iloc[train_idx]\n    y_train_fold = y_train_smote.iloc[train_idx]\n\n    X_valid_fold = X_train_smote.iloc[valid_idx]\n    y_valid_fold = y_train_smote.iloc[valid_idx]\n\n    # Initialize AdaBoost classifier\n    ada = AdaBoostClassifier(**params_adaboost)\n\n    # Train the AdaBoost model\n    ada.fit(X_train_fold, y_train_fold)\n    \n    fitted_models_ada.append(ada)\n\n    # Predict on the validation set\n    predictions = ada.predict_proba(X_valid_fold)[:, 1]\n\n    # Calculate AUC and store it\n    auc = roc_auc_score(y_valid_fold, predictions)  # Calculate AUC for this fold\n    auc_scores_adaboost.append(auc)\n\n    print(f\"Fold {fold_number + 1} AUC: {auc}\")\n    print(\"Maximum CV AUC score: \", max(auc_scores_adaboost))\n\n    # Check if this fold's AUC is the best\n    if auc > best_auc:\n        best_auc = auc\n        best_model = ada  # Store the best model\n\n# Save the best model to a file\nif best_model:\n    joblib.dump(best_model, model_filename)  # Save the best model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the AUC scores for each fold\nplt.plot(range(3), auc_scores_adaboost, marker='o', linestyle='-', color='b')\nplt.title(\"AUC for each Cross-Validation Fold\")\nplt.xlabel(\"Fold\")\nplt.ylabel(\"AUC\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import joblib\nfrom sklearn.metrics import roc_auc_score\n\n# Load the best model from the saved file\nbest_model = joblib.load(\"best_adaboost_model.joblib\")\n\n# Use the model to predict on the untrained data\npredictions_test = best_model.predict(X_test_smote)\n\n# Calculate AUC score on the test set\nauc_test = roc_auc_score(y_test_smote, predictions_test)\n\nprint(f\"Test AUC: {auc_test}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Evaluation with AUC and then comparison with the stability metric is shown below.","metadata":{"id":"Op9jsT2JhazU"}},{"cell_type":"code","source":"# for base, X in [(base_train, X_train), (base_valid, X_valid), (base_test, X_test)]:\n#     y_pred = gbm.predict(X, num_iteration=gbm.best_iteration)\n#     base[\"score\"] = y_pred\n\n# print(f'The AUC score on the train set is: {roc_auc_score(base_train[\"target\"], base_train[\"score\"])}')\n# print(f'The AUC score on the valid set is: {roc_auc_score(base_valid[\"target\"], base_valid[\"score\"])}')\n# print(f'The AUC score on the test set is: {roc_auc_score(base_test[\"target\"], base_test[\"score\"])}')","metadata":{"id":"Msjq0Qg2hazU","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def gini_stability(base, w_fallingrate=88.0, w_resstd=-0.5):\n#     gini_in_time = base.loc[:, [\"WEEK_NUM\", \"target\", \"score\"]]\\\n#         .sort_values(\"WEEK_NUM\")\\\n#         .groupby(\"WEEK_NUM\")[[\"target\", \"score\"]]\\\n#         .apply(lambda x: 2*roc_auc_score(x[\"target\"], x[\"score\"])-1).tolist()\n\n#     x = np.arange(len(gini_in_time))\n#     y = gini_in_time\n#     a, b = np.polyfit(x, y, 1)\n#     y_hat = a*x + b\n#     residuals = y - y_hat\n#     res_std = np.std(residuals)\n#     avg_gini = np.mean(gini_in_time)\n#     return avg_gini + w_fallingrate * min(0, a) + w_resstd * res_std\n\n# stability_score_train = gini_stability(base_train)\n# stability_score_valid = gini_stability(base_valid)\n# stability_score_test = gini_stability(base_test)\n\n# print(f'The stability score on the training set is: {stability_score_train}')\n# print(f'The stability score on the valid set is: {stability_score_valid}')\n# print(f'The stability score on the test set is: {stability_score_test}')","metadata":{"id":"xSczH1abhazU","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission\n\nScoring the submission dataset is below, we need to take care of new categories. Then we save the score as a last step.","metadata":{"id":"UGO8QrxbhazU"}},{"cell_type":"code","source":"from sklearn.base import BaseEstimator, RegressorMixin\nclass VotingModel(BaseEstimator, RegressorMixin):\n    def __init__(self, estimators):\n        super().__init__()\n        self.estimators = estimators\n        \n    def fit(self, X, y=None):\n        return self\n    \n    def predict(self, X):\n        y_preds = [estimator.predict(X) for estimator in self.estimators]\n        return np.mean(y_preds, axis=0)\n    \n    def predict_proba(self, X):\n        y_preds = [estimator.predict_proba(X) for estimator in self.estimators]\n        return np.mean(y_preds, axis=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = VotingModel(fitted_models_lgb+fitted_models_cat+fitted_models_hist)\nlen(model.estimators)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#combine X and Y\ndf_test = pd.concat([X_test_smote, y_test_smote], axis=1)\n\ndf_test = df_test.drop(columns=[\"WEEK_NUM\"])\ndf_test = df_test.set_index(\"case_id\")\ny_pred = pd.Series(model.predict_proba(df_test)[:, 1], index=df_test.index)\n\ndf_subm = pd.read_csv(ROOT / \"sample_submission.csv\")\ndf_subm = df_subm.set_index(\"case_id\")\n\ndf_subm[\"score\"] = y_pred\ndf_subm.to_csv(\"submission.csv\")\ndf_subm","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_submission = data_submission[cols_pred].to_pandas()\n# X_submission = convert_strings(X_submission)\n# categorical_cols = X_train.select_dtypes(include=['category']).columns\n\n# for col in categorical_cols:\n#     train_categories = set(X_train[col].cat.categories)\n#     submission_categories = set(X_submission[col].cat.categories)\n#     new_categories = submission_categories - train_categories\n#     X_submission.loc[X_submission[col].isin(new_categories), col] = \"Unknown\"\n#     new_dtype = pd.CategoricalDtype(categories=train_categories, ordered=True)\n#     X_train[col] = X_train[col].astype(new_dtype)\n#     X_submission[col] = X_submission[col].astype(new_dtype)\n\n# y_submission_pred = gbm.predict(X_submission, num_iteration=gbm.best_iteration)","metadata":{"id":"HBBUEUhMhazU","execution":{"iopub.status.busy":"2024-04-21T22:41:28.364152Z","iopub.status.idle":"2024-04-21T22:41:28.364987Z","shell.execute_reply.started":"2024-04-21T22:41:28.364685Z","shell.execute_reply":"2024-04-21T22:41:28.364709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission = pd.DataFrame({\n#     \"case_id\": data_submission[\"case_id\"].to_numpy(),\n#     \"score\": y_submission_pred\n# }).set_index('case_id')\n# submission.to_csv(\"./submission.csv\")","metadata":{"id":"o2WWjStahazU","execution":{"iopub.status.busy":"2024-04-21T22:41:28.366394Z","iopub.status.idle":"2024-04-21T22:41:28.366952Z","shell.execute_reply.started":"2024-04-21T22:41:28.366666Z","shell.execute_reply":"2024-04-21T22:41:28.36669Z"},"trusted":true},"execution_count":null,"outputs":[]}]}