{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50160,"databundleVersionId":7602123,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Created by yunsuxiaozi\n\n#### IMO,compared to playground, there may not be many people doing EDA in the code competition.Because in a code competition, the competition hoster will not provide ready-made features, and everyone needs to construct their own features. If EDA is disclosed, the constructed features will also be disclosed.\n\n#### This is my first time doing EDA in this competition.This competition has just begun, and my current ranking is not high. In addition, there may be a shake in the competition itself, so I am not worried about the loss of publishing this notebook. Beginners in the field of data mining are welcome to learn and exchange ideas.\n\n#### This is my current notebook for training and inference:\n\n#### <a href=\"https://www.kaggle.com/code/yunsuxiaozi/home-credit-baseline\">Home Credit baseline</a>\n\n#### <a href=\"https://www.kaggle.com/code/yunsuxiaozi/solution-of-threw-exception\">solution of Threw Exception</a>\n\n#### The features I construct here are very conventional. Open each dataset, if 'case_id' is unique, merge directly. If 'case_id' is not unique, calculate AGGREGATIONS, such as mean, std,min,max, and then merge.\n\n#### I have seen some notebooks, and EDA is just a decoration. After EDA is done, the original features are still thrown into the model for prediction. Therefore, I will explain how to change the features after EDA is done.\n\n#### Next, let's move on to the main text, where we first construct the features well.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import polars as pl#和pandas类似,但是处理大型数据集有更好的性能.\n#necessary\nimport pandas as pd#导入csv文件的库\nimport numpy as np#进行矩阵运算的库\n#model\nfrom lightgbm import LGBMClassifier\n#metric\nfrom sklearn.metrics import roc_auc_score#导入roc_auc曲线\n#KFold是直接分成k折,StratifiedKFold还要考虑每种类别的占比\nfrom sklearn.model_selection import StratifiedKFold\nimport dill#对对象进行序列化和反序列化(例如保存和加载树模型)\nimport gc#垃圾回收模块\n\n#config\nclass Config():\n    seed=2024\n    num_folds=10\n    TARGET_NAME ='target'\n    batch_size=1000#由于不知道测试数据的大小,所以分批次放入模型.\nimport random#提供了一些用于生成随机数的函数\n#设置随机种子,保证模型可以复现\ndef seed_everything(seed):\n    np.random.seed(seed)#numpy的随机种子\n    random.seed(seed)#python内置的随机种子\nseed_everything(Config.seed)\n\ndef set_table_dtypes(df: pl.DataFrame) -> pl.DataFrame:\n    # implement here all desired dtypes for tables\n    # the following is just an example\n    for col in df.columns:\n        # last letter of column name will help you determine the type\n        if col[-1] in (\"P\", \"A\"):\n            df = df.with_columns(pl.col(col).cast(pl.Float64).alias(col))\n\n    return df\ndef preprocessor(mode='train'):#mode='train'|'test'\n    #base 文件\n    print(\"base file\")\n    feats=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_base.csv\")\n            \n    print(\"applprev_2 file\")#这里保留最新的数据\n    applprev=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_applprev_2.csv\")\n    applprev=applprev.with_columns(pl.col('case_id').shift(-1).alias(\"case_id_shift_1\"))\n    #如果case_id不等于下一个case_id,说明它是最后一个数据了,保留下来\n    applprev=applprev.filter(pl.col('case_id')!=pl.col(\"case_id_shift_1\"))\n    applprev=applprev.drop([\"case_id_shift_1\"])\n    unique_value=['EMPLOYMENT_PHONE', 'PRIMARY_MOBILE', 'PRIMARY_EMAIL','PHONE','HOME_PHONE', 'SECONDARY_MOBILE', 'ALTERNATIVE_PHONE','WHATSAPP', 'SKYPE']\n    for value in unique_value:\n        applprev=applprev.with_columns((pl.col('conts_type_509L')==value).cast(pl.Int8).alias(f\"conts_type_509L_{value}\"))\n    feats=feats.join(applprev,on='case_id',how='left')\n    \n    del applprev\n    gc.collect()#手动触发垃圾回收,强制回收由垃圾回收器标记为未使用的内存\n    \n    print(\"credit_bureau file\")\n    credit_bureau_1=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_credit_bureau_b_1.csv\")\n    credit_bureau_2=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_credit_bureau_b_2.csv\")\n    \n    credit_bureau=credit_bureau_1.join(credit_bureau_2,on='case_id',how='left')\n    credit_bureau = credit_bureau.fill_null(-1)\n    cols=['amount_1115A', 'credlmt_1052A', 'credlmt_228A', 'credlmt_3940954A', 'credquantity_1099L', 'credquantity_984L', 'debtpastduevalue_732A', 'debtvalue_227A', 'dpd_550P', 'dpd_733P', 'dpdmax_851P', 'dpdmaxdatemonth_804T', 'dpdmaxdateyear_742T', 'installmentamount_644A', 'installmentamount_833A', 'instlamount_892A', 'maxdebtpduevalodued_3940955A', 'num_group1', 'numberofinstls_810L', 'overdueamountmax_950A', 'overdueamountmaxdatemonth_494T', 'overdueamountmaxdateyear_432T', 'pmtdaysoverdue_1135P', 'pmtnumpending_403L', 'residualamount_1093A', 'residualamount_127A', 'residualamount_3940956A', 'totalamount_503A', 'totalamount_881A', 'num_group1_right', 'num_group2', 'pmts_dpdvalue_108P', 'pmts_pmtsoverdue_635A']\n    #数值列的特征工程  从1开始是为了把'case_id'去掉    \n    for col in cols:\n        column_type = credit_bureau[col].dtype\n        is_numeric = (column_type == pl.datatypes.Int64) or (column_type == pl.datatypes.Float64) \n        if is_numeric:#数值列构造特征\n            feat=credit_bureau.group_by('case_id').agg( pl.max(col).alias(f\"max_credit_bureau_{col}\"),\n                                           pl.mean(col).alias(f\"mean_credit_bureau_{col}\"),\n                                           pl.median(col).alias(f\"median_credit_bureau_{col}\"),\n                                           pl.std(col).alias(f\"std_credit_bureau_{col}\"),\n                                           pl.min(col).alias(f\"min_credit_bureau_{col}\"),\n                                           pl.count(col).alias(f\"count_credit_bureau_{col}\"),\n                                           pl.sum(col).alias(f\"sum_credit_bureau_{col}\")\n                                         )\n            feats=feats.join(feat,on='case_id',how='left')\n    \n    del credit_bureau_1,credit_bureau_2,credit_bureau\n    gc.collect()#手动触发垃圾回收,强制回收由垃圾回收器标记为未使用的内存\n    \n    print(\"debitcard file\")\n    debitcard=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_debitcard_1.csv\")\n    ##数值列的特征工程  从1开始是为了把'case_id'去掉    \n    for idx in range(1,len(debitcard.columns)):\n        col=debitcard.columns[idx]\n        column_type = debitcard[col].dtype\n        is_numeric = (column_type == pl.datatypes.Int64) or (column_type == pl.datatypes.Float64) \n        if is_numeric:#数值列构造特征\n            feat=debitcard.group_by('case_id').agg( pl.max(col).alias(f\"max_debitcard_{col}\"),\n                                           pl.mean(col).alias(f\"mean_debitcard_{col}\"),\n                                           pl.median(col).alias(f\"median_debitcard_{col}\"),\n                                           pl.std(col).alias(f\"std_debitcard_{col}\"),\n                                           pl.min(col).alias(f\"min_debitcard_{col}\"),\n                                           pl.count(col).alias(f\"count_debitcard_{col}\"),\n                                           pl.sum(col).alias(f\"sum_debitcard_{col}\")\n                                         )\n            feats=feats.join(feat,on='case_id',how='left')\n    \n    del debitcard\n    gc.collect()#手动触发垃圾回收,强制回收由垃圾回收器标记为未使用的内存\n    \n    \n    #static_0,训练数据2个文件,测试数据3个文件（关于这里的处理,我自己处理总是会threw exception,所以抄https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook）\n    print(f\"static_0 file\")\n    #pipe用于在DataFrame上自定义自己的函数\n    static_0_0=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_static_0_0.csv\").pipe(set_table_dtypes)\n    static_0_1=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_static_0_1.csv\").pipe(set_table_dtypes)\n    \n    static=pl.concat([static_0_0,static_0_1],how=\"vertical_relaxed\")#垂直合并,并且放宽了数据类型匹配的限制\n    if mode=='test':#如果是测试数据的话还有一个文件\n        static_0_2=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_static_0_2.csv\").pipe(set_table_dtypes)\n        static=pl.concat([static,static_0_2],how=\"vertical_relaxed\")\n    feats=feats.join(static,on='case_id',how='left')\n    del static,static_0_0,static_0_1\n    gc.collect()#手动触发垃圾回收,强制回收由垃圾回收器标记为未使用的内存\n    \n    #static_cb文件\n    print(\"static_cb_file\")\n    static_cb=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_static_cb_0.csv\")\n    feats=feats.join(static_cb,on='case_id',how='left')\n    del static_cb\n    gc.collect()#手动触发垃圾回收,强制回收由垃圾回收器标记为未使用的内存\n    \n    #tax的3个文件(tax_c这个文件好像存在些问题,暂时不放入训练数据)\n    print(\"tax_a,b,c_file\")\n    tax_a=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_tax_registry_a_1.csv\")\n    tax_b=pl.read_csv(f\"/kaggle/input/home-credit-credit-risk-model-stability/csv_files/{mode}/{mode}_tax_registry_b_1.csv\")\n    tax=tax_a.join(tax_b,on=['case_id','num_group1'],how='left')\n    #数值列的特征工程  从1开始是为了把'case_id'去掉    \n    for idx in range(1,len(tax.columns)):\n        col=tax.columns[idx]\n        column_type = tax[col].dtype\n        is_numeric = (column_type == pl.datatypes.Int64) or (column_type == pl.datatypes.Float64) \n        if is_numeric:#数值列构造特征\n            feat=tax.group_by('case_id').agg( pl.max(col).alias(f\"max_tax_{col}\"),\n                                           pl.mean(col).alias(f\"mean_tax_{col}\"),\n                                           pl.median(col).alias(f\"median_tax_{col}\"),\n                                           pl.std(col).alias(f\"std_tax_{col}\"),\n                                           pl.min(col).alias(f\"min_tax_{col}\"),\n                                           pl.count(col).alias(f\"count_tax_{col}\"),\n                                           pl.sum(col).alias(f\"sum_tax_{col}\")\n                                         )\n            feats=feats.join(feat,on='case_id',how='left')\n    del tax_a,tax_b,tax\n    gc.collect()#手动触发垃圾回收,强制回收由垃圾回收器标记为未使用的内存\n    \n    #对month和weeknum的处理\n    feats=feats.with_columns((pl.col('MONTH')%100).alias(\"MONTH\"))\n    feats=feats.with_columns((pl.col('WEEK_NUM')%12).alias(\"WEEK_NUM\"))\n    \n    return feats\n\ntrain_feats=preprocessor(mode='train')\nprint(f\"len(train_feats):{len(train_feats)}\")\n\ntest_feats=preprocessor(mode='test')\nprint(f\"len(test_feats):{len(test_feats)}\")\n\n#如果打开的两个文件的相同列一个是浮点数类型,一个是object或者str,就把两个都转成浮点数类型.\nfor col in test_feats.columns:\n    if (train_feats[col].dtype==pl.datatypes.Float64) or (test_feats[col].dtype==pl.datatypes.Float64):\n        train_feats.with_columns(train_feats[col].cast(pl.datatypes.Float64))\n        test_feats.with_columns(test_feats[col].cast(pl.datatypes.Float64))\ntrain_feats=train_feats.to_pandas()\ntest_feats=test_feats.to_pandas()\n#如果是字符串的列或者一列只有唯一值,去掉\ndrop_cols=[]\nfor col in test_feats.columns:\n    if (train_feats[col].dtype=='object') or (test_feats[col].dtype=='object') or (train_feats[col].nunique()==1):\n        drop_cols+=[col]\ndrop_cols+=['case_id']\nprint(f\"len(drop_cols):{len(drop_cols)},drop_cols:{drop_cols}\")\ntrain_feats=train_feats.drop(drop_cols,axis=1)\ntest_feats=test_feats.drop(drop_cols,axis=1)\nprint(\"fillna\")\ntrain_feats.fillna(-1,inplace=True)\ntest_feats.fillna(-1,inplace=True)\nprint(f\"len(drop_cols):{len(drop_cols)},total_features_count:{len(test_feats.columns)}\")\ntrain_feats.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Sample deduplication not only reduces memory, but also prevents training data from appearing in validation data during cross validation.\n\n#### In my opinion, week_num is not an important feature because the week_num in the test data is larger than the week_num in the training data. Although I have calculated the week_num for a quarter here.(week_num% 12),I found that their mean values are very close.Let's see how many samples are duplicated.","metadata":{}},{"cell_type":"code","source":"#去掉时间列和target列\ntrain_feats_copy=train_feats.drop(['WEEK_NUM'],axis=1).copy()\nprint(f\"len(train_feats_copy):{len(train_feats_copy)}\")\ntrain_feats_copy.drop_duplicates(inplace=True)\nprint(f\"len(train_feats_copy):{len(train_feats_copy)}\")\ntrain_feats['target'].groupby(train_feats['WEEK_NUM']).mean().reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### We have determined that week_num is a useless feature, and we can also confirm month in the same way next.The feature of MONTH is slightly better than that of week_num.Based on the above analysis, we have decided to retain MONTH.","metadata":{}},{"cell_type":"code","source":"train_feats['target'].groupby(train_feats['MONTH']).mean().reset_index()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### A situation may arise in time series where the same features have different targets at different times. We can take a look here.We can find that the data that has been deduplicated is less than the data that has been deduplicated above.","metadata":{}},{"cell_type":"code","source":"train_feats_copy=train_feats.drop(['WEEK_NUM','target'],axis=1).copy()\nprint(f\"len(train_feats_copy):{len(train_feats_copy)}\")\ntrain_feats_copy.drop_duplicates(inplace=True)\nprint(f\"len(train_feats_copy):{len(train_feats_copy)}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Find 1:  Based on the EDA mentioned above, my suggestion is to remove the feature week_num and retain the feature with a later time when faced with the same feature but different targets.\n\n####  This  is a binary classification task. We definitely hope to find some categorical variables. When a categorical variable has a certain value, the target is always 0 or 1. If this pattern is reliable, we can fine tune the model's prediction results.The pattern I am referring to here is reliable, it cannot have exactly one data, at least 100 data.\n","metadata":{}},{"cell_type":"code","source":"#这里想看看类别型变量中有没有\nfor col in train_feats.columns: \n    if train_feats[col].nunique()<100 and col!='target':#如果是类别型变量\n        #当类别型变量为某个值时,target的均值\n        tmp_df=train_feats['target'].groupby(train_feats[col]).mean().reset_index()\n        #找到均值为0或者均值为1的,也就是当类别型变量为某个值时,target是确定的值\n        tmp_df=tmp_df[(tmp_df['target']==0)|(tmp_df['target']==1)]\n        if len(tmp_df):#如果表格中有数据,就遍历输出\n            x,y=tmp_df[col].values,tmp_df['target'].values\n            for idx in range(len(x)):\n                #看看类别型变量为这个值的时候数据量有多少\n                data_count=len(train_feats[train_feats[col]==x[idx]])\n                if data_count>100:#数据量有100个说明相对可信\n                    print(f\"when {col}={x[idx]},target={y[idx]},data_count:{data_count}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Find 2:We can fine tune the model based on these determined targets.\n\n#### We need to check the correlation of the constructed features. Due to the large number of files here, it is possible to include the same variables from different files in the features at the same time, or there may be some highly correlated features. For these highly correlated features, dimensionality reduction can be considered, which can save memory space and improve model performance.","metadata":{}},{"cell_type":"code","source":"#我们这里就是找相关性特别高的特征对,可以考虑对它们进行降维操作.\n#计算两组变量的皮尔逊相关系数\ndef pearson_corr(x1,x2):\n    \"\"\"\n    x1,x2:np.array\n    \"\"\"\n    mean_x1=np.mean(x1)\n    mean_x2=np.mean(x2)\n    std_x1=np.std(x1)\n    std_x2=np.std(x2)\n    pearson=np.mean((x1-mean_x1)*(x2-mean_x2))/(std_x1*std_x2)\n    return pearson\ndef find_corr():\n    cols=train_feats.columns\n    drop_cols=[]#2个特征相关性系数非常接近1了,所以会考虑drop掉一个特征\n    corr_cols=[]#比如第1,3,4个特征它们的相关性特别高,我们会考虑[[1,3,4]],然后降维\n    ignore_cols=[]#在遍历过程中出现在idx后面的特征,可能已经和idx前面的某个特征相关性高,需要做降维了,所以不需要再计算相关性了\n    for idx in range(len(cols)):\n        if cols[idx] in ignore_cols:\n            continue\n        corr_col=[cols[idx]]#这里和idx相关的特征\n        for j in range(idx+1,len(cols)):\n            if cols[j] in ignore_cols:\n                continue\n            #计算两组特征的皮尔逊相关系数\n            pearson=pearson_corr(train_feats[cols[idx]].values,train_feats[cols[j]].values)\n            if abs(pearson)>0.99:#相关性太高了,直接去掉一个特征\n                drop_cols.append(cols[j])\n                ignore_cols.append(cols[j])\n                print(f\"the {cols[idx]} and {cols[j]} pearson_corr is {pearson}\")\n            elif abs(pearson)>0.8:#如果皮尔逊相关系数一般高\n                corr_col.append(cols[j])\n                ignore_cols.append(cols[j])\n                print(f\"the {cols[idx]} and {cols[j]} pearson_corr is {pearson}\")\n        if len(corr_col)>1:#因为已经有cols[idx],大于1就是有其他相关性特征.\n            corr_cols.append(corr_col)\n            print(\"-\"*50)\n    #因为和某些特征相关性为1,所以去掉的列名\n    print(f\"drop_cols:{drop_cols}\")\n    print(f\"pearson_corr_pairs:{len(corr_cols)}\")\n    for idx in range(len(corr_cols)):\n        print(f\"pairs_{idx}:{corr_cols[idx]}\")\nfind_corr()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Find3: We can remove drop_cols and find ways to reduce the dimensionality of the features in corr_cols.\n\n#### I am currently only thinking of these. If you have any good ideas, please feel free to communicate with me. If you think my notebook is useful, please give me a vote.","metadata":{}}]}