{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7921029,"sourceType":"competition"}],"dockerImageVersionId":30684,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.19"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Tabnet notebook for beginner🔰\n  \n<div class=\"alert alert-block alert-warning\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    📌 &nbsp; In this notebook, I will introduce Tabnet. What is Tabnet ? How to use?\n</div>\n","metadata":{}},{"cell_type":"markdown","source":"## What is Tabnet ?? \n\nTabNet is a modern Neural Network architecture for tabular data.It is specifically made to adequately manage all those situations in which we work on data in tabular format.It was presented at the end of 2020 and its best description can be found in the article available on the usual ArXiv: https://arxiv.org/pdf/1908.07442.pdf\n\nWe know that neural networks give exceptional results on unstructured data, such as in the field of Image Recognition and Object Detection (Images), as in the field of Natural Language Processing (texts), in “Speech-to-text”, often proving to be able to reach and overcome the “Human-Level Performance”. But the generalized idea, until now, is that on tabular data it is often easier and faster to obtain results using algorithms not based on Neural Networks, such as Gradient Boosting algorithms. In fact, it is sufficient to analyze the competition leaderboards on Kaggle for these types of data: the best results are obtained using implementations, which can also run quickly on GPUs, such as XGBoost, LightGBM, CatBoost.","metadata":{}},{"cell_type":"markdown","source":"## Import packege","metadata":{}},{"cell_type":"code","source":"# !pip install pytorch-tabnet","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nfrom pathlib import Path\nimport subprocess\nimport os\nimport gc\nfrom glob import glob\n\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nfrom datetime import datetime\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nROOT = '/kaggle/input/home-credit-credit-risk-model-stability'","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2024-04-10T13:06:25.153522Z","iopub.status.busy":"2024-04-10T13:06:25.153052Z","iopub.status.idle":"2024-04-10T13:06:28.709964Z","shell.execute_reply":"2024-04-10T13:06:28.708451Z","shell.execute_reply.started":"2024-04-10T13:06:25.153489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import TimeSeriesSplit, GroupKFold, StratifiedGroupKFold\nfrom sklearn.base import BaseEstimator, RegressorMixin\nfrom sklearn.metrics import roc_auc_score\nimport lightgbm as lgb\nfrom pytorch_tabnet.tab_model import TabNetClassifier, TabNetRegressor","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:28.713322Z","iopub.status.busy":"2024-04-10T13:06:28.712497Z","iopub.status.idle":"2024-04-10T13:06:30.810851Z","shell.execute_reply":"2024-04-10T13:06:30.809110Z","shell.execute_reply.started":"2024-04-10T13:06:28.713275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Define class and function","metadata":{}},{"cell_type":"code","source":"class Pipeline:\n\n    def set_table_dtypes(df):\n        for col in df.columns:\n            if col in [\"case_id\", \"WEEK_NUM\", \"num_group1\", \"num_group2\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Int64))\n            elif col in [\"date_decision\"]:\n                df = df.with_columns(pl.col(col).cast(pl.Date))\n            elif col[-1] in (\"P\", \"A\"):\n                df = df.with_columns(pl.col(col).cast(pl.Float64))\n            elif col[-1] in (\"M\",):\n                df = df.with_columns(pl.col(col).cast(pl.String))\n            elif col[-1] in (\"D\",):\n                df = df.with_columns(pl.col(col).cast(pl.Date))\n        return df\n\n    def handle_dates(df):\n        for col in df.columns:\n            if col[-1] in (\"D\",):\n                df = df.with_columns(pl.col(col) - pl.col(\"date_decision\"))  #!!?\n                df = df.with_columns(pl.col(col).dt.total_days()) # t - t-1\n        df = df.drop(\"date_decision\", \"MONTH\")\n        return df\n\n    def filter_cols(df):\n        for col in df.columns:\n            if col not in [\"target\", \"case_id\", \"WEEK_NUM\"]:\n                isnull = df[col].is_null().mean()\n                if isnull > 0.3:\n                    df = df.drop(col)\n        \n        for col in df.columns:\n            if (col not in [\"target\", \"case_id\", \"WEEK_NUM\"]) & (df[col].dtype == pl.String):\n                freq = df[col].n_unique()\n                if (freq == 1) | (freq > 200):\n                    df = df.drop(col)\n        \n        return df","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:37.235646Z","iopub.status.busy":"2024-04-10T13:06:37.234827Z","iopub.status.idle":"2024-04-10T13:06:37.253171Z","shell.execute_reply":"2024-04-10T13:06:37.251443Z","shell.execute_reply.started":"2024-04-10T13:06:37.235596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nclass Aggregator:\n    #Please add or subtract features yourself, be aware that too many features will take up too much space.\n    def num_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"P\", \"A\")]\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        \n        expr_last = [pl.last(col).alias(f\"last_{col}\") for col in cols]\n        #expr_first = [pl.first(col).alias(f\"first_{col}\") for col in cols]\n        expr_mean = [pl.mean(col).alias(f\"mean_{col}\") for col in cols]\n        expr_var = [pl.var(col).alias(f\"var_{col}\") for col in cols]\n\n        return expr_max +expr_last+expr_mean+expr_var\n    \n    def date_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"D\")]\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        #expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n        expr_last = [pl.last(col).alias(f\"last_{col}\") for col in cols]\n        #expr_first = [pl.first(col).alias(f\"first_{col}\") for col in cols]\n        expr_mean = [pl.mean(col).alias(f\"mean_{col}\") for col in cols]\n        return  expr_max +expr_last+expr_mean\n    \n    def str_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"M\",)]\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        #expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n        expr_last = [pl.last(col).alias(f\"last_{col}\") for col in cols]\n        #expr_first = [pl.first(col).alias(f\"first_{col}\") for col in cols]\n        #expr_count = [pl.count(col).alias(f\"count_{col}\") for col in cols]\n        return  expr_max +expr_last#+expr_count\n    \n    def other_expr(df):\n        cols = [col for col in df.columns if col[-1] in (\"T\", \"L\")]\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols]\n        #expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n        expr_last = [pl.last(col).alias(f\"last_{col}\") for col in cols]\n        #expr_first = [pl.first(col).alias(f\"first_{col}\") for col in cols]\n        return  expr_max +expr_last\n    \n    def count_expr(df):\n        cols = [col for col in df.columns if \"num_group\" in col]\n        expr_max = [pl.max(col).alias(f\"max_{col}\") for col in cols] \n        #expr_min = [pl.min(col).alias(f\"min_{col}\") for col in cols]\n        expr_last = [pl.last(col).alias(f\"last_{col}\") for col in cols]\n        #expr_first = [pl.first(col).alias(f\"first_{col}\") for col in cols]\n        return  expr_max +expr_last\n    \n    def get_exprs(df):\n        exprs = Aggregator.num_expr(df) + \\\n                Aggregator.date_expr(df) + \\\n                Aggregator.str_expr(df) + \\\n                Aggregator.other_expr(df) + \\\n                Aggregator.count_expr(df)\n\n        return exprs","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:37.861556Z","iopub.status.busy":"2024-04-10T13:06:37.860795Z","iopub.status.idle":"2024-04-10T13:06:37.880744Z","shell.execute_reply":"2024-04-10T13:06:37.879644Z","shell.execute_reply.started":"2024-04-10T13:06:37.861515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_file(path, depth=None):\n    df = pl.read_parquet(path)\n    df = df.pipe(Pipeline.set_table_dtypes)\n    if depth in [1,2]:\n        df = df.group_by(\"case_id\").agg(Aggregator.get_exprs(df)) \n    return df\n\ndef read_files(regex_path, depth=None):\n    chunks = []\n    \n    for path in glob(str(regex_path)):\n        df = pl.read_parquet(path)\n        df = df.pipe(Pipeline.set_table_dtypes)\n        if depth in [1, 2]:\n            df = df.group_by(\"case_id\").agg(Aggregator.get_exprs(df))\n        chunks.append(df)\n    \n    df = pl.concat(chunks, how=\"vertical_relaxed\")\n    df = df.unique(subset=[\"case_id\"])\n    return df","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:38.251954Z","iopub.status.busy":"2024-04-10T13:06:38.250554Z","iopub.status.idle":"2024-04-10T13:06:38.263023Z","shell.execute_reply":"2024-04-10T13:06:38.261120Z","shell.execute_reply.started":"2024-04-10T13:06:38.251882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def feature_eng(df_base, depth_0, depth_1, depth_2):\n    df_base = (\n        df_base\n        .with_columns(\n            month_decision = pl.col(\"date_decision\").dt.month(),\n            weekday_decision = pl.col(\"date_decision\").dt.weekday(),\n        )\n    )\n    for i, df in enumerate(depth_0 + depth_1 + depth_2):\n        df_base = df_base.join(df, how=\"left\", on=\"case_id\", suffix=f\"_{i}\")\n    df_base = df_base.pipe(Pipeline.handle_dates)\n    return df_base","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:38.704430Z","iopub.status.busy":"2024-04-10T13:06:38.703767Z","iopub.status.idle":"2024-04-10T13:06:38.713112Z","shell.execute_reply":"2024-04-10T13:06:38.711178Z","shell.execute_reply.started":"2024-04-10T13:06:38.704373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def to_pandas(df_data, cat_cols=None):\n    df_data = df_data.to_pandas()\n    if cat_cols is None:\n        cat_cols = list(df_data.select_dtypes(\"object\").columns)\n    df_data[cat_cols] = df_data[cat_cols].astype(\"category\")\n    return df_data, cat_cols","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:39.357106Z","iopub.status.busy":"2024-04-10T13:06:39.355868Z","iopub.status.idle":"2024-04-10T13:06:39.363570Z","shell.execute_reply":"2024-04-10T13:06:39.362137Z","shell.execute_reply.started":"2024-04-10T13:06:39.357058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def reduce_mem_usage(df):\n    \"\"\" iterate through all the columns of a dataframe and modify the data type\n        to reduce memory usage.        \n    \"\"\"\n    start_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        if str(col_type)==\"category\":\n            continue\n        \n        if col_type != object:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n        else:\n            continue\n    end_mem = df.memory_usage().sum() / 1024**2\n    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n    \n    return df","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:39.697986Z","iopub.status.busy":"2024-04-10T13:06:39.696984Z","iopub.status.idle":"2024-04-10T13:06:39.716629Z","shell.execute_reply":"2024-04-10T13:06:39.715140Z","shell.execute_reply.started":"2024-04-10T13:06:39.697935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def reduce_group(grps):\n    use = []\n    for g in grps:\n        mx = 0; vx = g[0]\n        for gg in g:\n            n = df_train[gg].nunique()\n            if n>mx:\n                mx = n\n                vx = gg\n            #print(str(gg)+'-'+str(n),', ',end='')\n        use.append(vx)\n        #print()\n    print('Use these',use)\n    return use\n\ndef group_columns_by_correlation(matrix, threshold=0.7):\n    correlation_matrix = matrix.corr()\n\n    # 分组列\n    groups = []\n    remaining_cols = list(matrix.columns)\n    while remaining_cols:\n        col = remaining_cols.pop(0)\n        group = [col]\n        correlated_cols = [col]\n        for c in remaining_cols:\n            if correlation_matrix.loc[col, c] >= threshold:\n                group.append(c)\n                correlated_cols.append(c)\n        groups.append(group)\n        remaining_cols = [c for c in remaining_cols if c not in correlated_cols]\n    \n    return groups","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Loading and Feature Engineering","metadata":{}},{"cell_type":"code","source":"ROOT            = Path(\"/kaggle/input/home-credit-credit-risk-model-stability\")\nROOT            = Path(\"./home-credit-credit-risk-model-stability\")\n\nTRAIN_DIR       = ROOT / \"parquet_files\" / \"train\"\nTEST_DIR        = ROOT / \"parquet_files\" / \"test\"","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:39.977535Z","iopub.status.busy":"2024-04-10T13:06:39.976669Z","iopub.status.idle":"2024-04-10T13:06:39.984127Z","shell.execute_reply":"2024-04-10T13:06:39.982998Z","shell.execute_reply.started":"2024-04-10T13:06:39.977480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_store = {\n    \"df_base\": read_file(TRAIN_DIR / \"train_base.parquet\"),\n    \"depth_0\": [\n        read_file(TRAIN_DIR / \"train_static_cb_0.parquet\"),\n        read_files(TRAIN_DIR / \"train_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        read_files(TRAIN_DIR / \"train_applprev_1_*.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_a_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_b_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_tax_registry_c_1.parquet\", 1),\n        read_files(TRAIN_DIR / \"train_credit_bureau_a_1_*.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_other_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_person_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_deposit_1.parquet\", 1),\n        read_file(TRAIN_DIR / \"train_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        read_file(TRAIN_DIR / \"train_credit_bureau_b_2.parquet\", 2),\n        read_files(TRAIN_DIR / \"train_credit_bureau_a_2_*.parquet\", 2),\n        read_file(TRAIN_DIR / \"train_applprev_2.parquet\", 2),\n        read_file(TRAIN_DIR / \"train_person_2.parquet\", 2)\n    ]\n}","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:06:40.939551Z","iopub.status.busy":"2024-04-10T13:06:40.938089Z","iopub.status.idle":"2024-04-10T13:10:24.268767Z","shell.execute_reply":"2024-04-10T13:10:24.266756Z","shell.execute_reply.started":"2024-04-10T13:06:40.939488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = feature_eng(**data_store)\nprint(\"train data shape:\\t\", df_train.shape)\ndel data_store\ndf_train = df_train.pipe(Pipeline.filter_cols)\nprint(\"train data shape:\\t\", df_train.shape)\ngc.collect()","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:10:24.276267Z","iopub.status.busy":"2024-04-10T13:10:24.275739Z","iopub.status.idle":"2024-04-10T13:11:06.056805Z","shell.execute_reply":"2024-04-10T13:11:06.055594Z","shell.execute_reply.started":"2024-04-10T13:10:24.276226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train, cat_cols = to_pandas(df_train)\ndf_train = reduce_mem_usage(df_train)\nprint(\"train data shape:\\t\", df_train.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nums=df_train.select_dtypes(exclude='category').columns\nfrom itertools import combinations, permutations\n#df_train=df_train[nums]\nnans_df = df_train[nums].isna()\nnans_groups={}\nfor col in nums:\n    cur_group = nans_df[col].sum()\n    try:\n        nans_groups[cur_group].append(col)\n    except:\n        nans_groups[cur_group]=[col]\ndel nans_df; x=gc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"uses=[]\nfor k,v in nans_groups.items():\n    if len(v)>1:\n            Vs = nans_groups[k]\n            #cross_features=list(combinations(Vs, 2))\n            #make_corr(Vs)\n            grps= group_columns_by_correlation(df_train[Vs], threshold=0.8)\n            use=reduce_group(grps)\n            uses=uses+use\n            #make_corr(use)\n    else:\n        uses=uses+v\n    print('####### NAN count =',k)\nprint(uses)\nprint(len(uses))\nuses=uses+list(df_train.select_dtypes(include='category').columns)\nprint(len(uses))\ndf_train=df_train[uses]\n# df_train.drop(['requesttype_4525192L_cnt','max_empl_employedtotal_800L_cnt', 'max_empl_industry_691L_cnt'], axis=1, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:11:40.722152Z","iopub.status.busy":"2024-04-10T13:11:40.721772Z","iopub.status.idle":"2024-04-10T13:15:21.255127Z","shell.execute_reply":"2024-04-10T13:15:21.253555Z","shell.execute_reply.started":"2024-04-10T13:11:40.722108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sample = pd.read_csv(\"/kaggle/input/home-credit-credit-risk-model-stability/sample_submission.csv\")\nsample = pd.read_csv(\"./home-credit-credit-risk-model-stability/sample_submission.csv\")\ndevice='gpu'\n#n_samples=200000\nDRY_RUN = True if sample.shape[0] == 10 else False   \nif DRY_RUN:\n    device='cpu'\n    # df_train = df_train.iloc[:50000]\nprint(device)","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:21.257507Z","iopub.status.busy":"2024-04-10T13:15:21.257057Z","iopub.status.idle":"2024-04-10T13:15:21.276131Z","shell.execute_reply":"2024-04-10T13:15:21.274570Z","shell.execute_reply.started":"2024-04-10T13:15:21.257469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_store = {\n    \"df_base\": read_file(TEST_DIR / \"test_base.parquet\"),\n    \"depth_0\": [\n        read_file(TEST_DIR / \"test_static_cb_0.parquet\"),\n        read_files(TEST_DIR / \"test_static_0_*.parquet\"),\n    ],\n    \"depth_1\": [\n        read_files(TEST_DIR / \"test_applprev_1_*.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_a_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_b_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_tax_registry_c_1.parquet\", 1),\n        read_files(TEST_DIR / \"test_credit_bureau_a_1_*.parquet\", 1),\n        read_file(TEST_DIR / \"test_credit_bureau_b_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_other_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_person_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_deposit_1.parquet\", 1),\n        read_file(TEST_DIR / \"test_debitcard_1.parquet\", 1),\n    ],\n    \"depth_2\": [\n        read_file(TEST_DIR / \"test_credit_bureau_b_2.parquet\", 2),\n        read_files(TEST_DIR / \"test_credit_bureau_a_2_*.parquet\", 2),\n        read_file(TEST_DIR / \"test_applprev_2.parquet\", 2),\n        read_file(TEST_DIR / \"test_person_2.parquet\", 2)\n    ]\n}","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:21.278151Z","iopub.status.busy":"2024-04-10T13:15:21.277758Z","iopub.status.idle":"2024-04-10T13:15:21.673540Z","shell.execute_reply":"2024-04-10T13:15:21.672272Z","shell.execute_reply.started":"2024-04-10T13:15:21.278118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = feature_eng(**data_store)\nprint(\"test data shape:\\t\", df_test.shape)\ndel data_store\ngc.collect()","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:21.677372Z","iopub.status.busy":"2024-04-10T13:15:21.676283Z","iopub.status.idle":"2024-04-10T13:15:21.968323Z","shell.execute_reply":"2024-04-10T13:15:21.966978Z","shell.execute_reply.started":"2024-04-10T13:15:21.677329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ndf_test = df_test.select([col for col in df_train.columns if col not in ['requesttype_4525192L_cnt',\n                                                                         'max_empl_employedtotal_800L_cnt',\n                                                                         'max_empl_industry_691L_cnt',\n                                                                         \"target\"]])\nprint(\"train data shape:\\t\", df_train.shape)\nprint(\"test data shape:\\t\", df_test.shape)\n\ndf_test, cat_cols = to_pandas(df_test, cat_cols)\ndf_test = reduce_mem_usage(df_test)\n\ngc.collect()","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:22.205934Z","iopub.status.busy":"2024-04-10T13:15:22.205350Z","iopub.status.idle":"2024-04-10T13:15:22.678955Z","shell.execute_reply":"2024-04-10T13:15:22.677717Z","shell.execute_reply.started":"2024-04-10T13:15:22.205881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"convert_cols = ['month_decision', 'weekday_decision']\n\ndf_train[convert_cols] = df_train[convert_cols].astype('category')\ndf_test[convert_cols] = df_test[convert_cols].astype('category')","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:22.775490Z","iopub.status.busy":"2024-04-10T13:15:22.774596Z","iopub.status.idle":"2024-04-10T13:15:22.803500Z","shell.execute_reply":"2024-04-10T13:15:22.801911Z","shell.execute_reply.started":"2024-04-10T13:15:22.775416Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def encode(df):\n    try:\n        return np.cos(2 * np.pi * df / 12), np.sin(2 * np.pi * df / 12)\n    except:\n        return np.NaN, np.NaN\n        \ndef encoding_timeseries_to_SinCos(df, col):\n    array = df[col].to_numpy()\n    encoded = np.ndarray((len(df), 2))\n\n    for i in trange(len(array)):\n        num = encode(array[i])\n        encoded[i] = num\n        del num\n    return encoded ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tqdm.notebook import trange\ndf_train[['month_decision_cos', 'month_decision_sin']] = encoding_timeseries_to_SinCos(df_train, 'month_decision')\ndf_train[['weekday_decision_cos', 'weekday_decision_sin']] = encoding_timeseries_to_SinCos(df_train, 'weekday_decision')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test[['month_decision_cos', 'month_decision_sin']] = encoding_timeseries_to_SinCos(df_test, 'month_decision')\ndf_test[['weekday_decision_cos', 'weekday_decision_sin']] = encoding_timeseries_to_SinCos(df_test, 'weekday_decision')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preproccessing for Tabnet\n\nNumerical conversion and null completion of categorical features are required in advance.  \nIn the example implementation, the categorical features are identified by data type and number of uniques, which should be changed as needed.","metadata":{}},{"cell_type":"code","source":"df_train['opencred_647L']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfrom tqdm.notebook import tqdm\n# Simple preprocessing\nnunique = df_train.nunique()\ntypes = df_train.dtypes\ndf_train[cat_cols] = df_train[cat_cols].astype('str')\ndf_test[cat_cols] = df_test[cat_cols].astype('str')\n\ncategorical_columns = []\ncategorical_dims =  {}\nfor col in tqdm(df_train.columns, total=len(df_train.columns)):\n    # print(types[col])\n    if types[col] == 'object' or types[col] == 'category':\n        # print(col, df_train[col].nunique())\n        l_enc = LabelEncoder()\n        df_train[col] = df_train[col].fillna(\"VV_likely\")\n        df_train[col] = l_enc.fit_transform(df_train[col].values)\n        categorical_columns.append(col)\n        categorical_dims[col] = len(l_enc.classes_)\n    elif col in ['MONTH', 'WEEK_NUM', 'target']:\n        continue\n    else:\n        # print(col)\n        # df_train.fillna(df_train[col].mean(), inplace=True) \n        continue\n\n# Define categorical features for categorical embeddings\nfeatures = [col for col in df_train.columns if col not in ['target']] \ncat_idxs = [i for i, f in enumerate(features) if f in categorical_columns]\ncat_dims = [categorical_dims[f] for i, f in enumerate(features) if f in categorical_columns]\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DRY_RUN = True if sample.shape[0] == 10 else False   \nif DRY_RUN:\n    device='CPU'\n    df_train = df_train.iloc[:25000]\n    #n_samples=10000","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = df_train[\"target\"]\nweeks = df_train[\"WEEK_NUM\"]\ndf_train= df_train.drop(columns=[\"target\", \"case_id\", \"WEEK_NUM\"])\ncv = StratifiedGroupKFold(n_splits=5, shuffle=False)","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:22.806025Z","iopub.status.busy":"2024-04-10T13:15:22.805317Z","iopub.status.idle":"2024-04-10T13:15:22.961702Z","shell.execute_reply":"2024-04-10T13:15:22.960714Z","shell.execute_reply.started":"2024-04-10T13:15:22.805981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_numpy = df_train.fillna(-9999).to_numpy()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_numpy = np.array(y)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clf = TabNetClassifier()  #TabNetRegressor()\n\nfor idx_train, idx_valid in cv.split(df_train, y, groups=weeks):#\n    X_train, y_train = df_train_numpy[idx_train], y_numpy[idx_train]\n    X_valid, y_valid = df_train_numpy[idx_valid], y_numpy[idx_valid]\n    clf.fit(\n    X_train, y_train,\n    eval_set=[(X_valid, y_valid)]\n    )\n    break","metadata":{"execution":{"iopub.execute_input":"2024-04-10T13:15:22.963981Z","iopub.status.busy":"2024-04-10T13:15:22.963524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"df_test = df_test.drop(columns=[\"WEEK_NUM\"])\ndf_test = df_test.set_index(\"case_id\")\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_columns = []\ncategorical_dims =  {}\nfor col in tqdm(df_test.columns, total=len(df_test.columns)):\n    # print(types[col])\n    if types[col] == 'object' or types[col] == 'category':\n        # print(col, df_test[col].nunique())\n        l_enc = LabelEncoder()\n        df_test[col] = df_test[col].fillna(\"VV_likely\")\n        df_test[col] = l_enc.fit_transform(df_test[col].values)\n        categorical_columns.append(col)\n        categorical_dims[col] = len(l_enc.classes_)\n    elif col in ['MONTH', 'WEEK_NUM', 'target']:\n        continue\n    else:\n        # print(col)\n        # df_test.fillna(df_test[col].mean(), inplace=True) \n        continue\n\n# Define categorical features for categorical embeddings\nfeatures = [col for col in df_test.columns if col not in ['target']] \ncat_idxs = [i for i, f in enumerate(features) if f in categorical_columns]\ncat_dims = [categorical_dims[f] for i, f in enumerate(features) if f in categorical_columns]\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test_numpy = df_test.fillna(-9999).to_numpy()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clf.predict_proba(df_test_numpy)[:, 1]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = pd.Series(clf.predict_proba(df_test_numpy)[:, 1], index=df_test.index)\ndf_subm = pd.read_csv(ROOT / \"sample_submission.csv\")\ndf_subm = df_subm.set_index(\"case_id\")\ndf_subm['score'] = y_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_csv(\"./home-credit-credit-risk-model-stability/csv_files/test/test_base.csv\")\n# test = pd.read_csv(\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/test/test_base.csv\")\nsub = pd.read_csv(\"./home-credit-credit-risk-model-stability/sample_submission.csv\", dtype={\"case_id\": int}).set_index(\"case_id\")\n# sub = pd.read_csv(\"/kaggle/input/home-credit-credit-risk-model-stability/sample_submission.csv\", dtype={\"case_id\": int}).set_index(\"case_id\")\nSHIFT = 0.025\nweek_num = list(test[\"WEEK_NUM\"])\ndf_subm[\"WEEK_NUM\"] = week_num\ncondition = df_subm[\"WEEK_NUM\"] < (df_subm[\"WEEK_NUM\"].max() - df_subm[\"WEEK_NUM\"].min())/2 + df_subm[\"WEEK_NUM\"].min()\ndf_subm.loc[condition, 'score'] = (df_subm.loc[condition, 'score'] - SHIFT).clip(0)\ndel df_subm[\"WEEK_NUM\"]\ndf_subm.to_csv(\"submission.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}