\n EDA. Part I: Nulls, zeros, outliers... Somthing else?\n
\n \n Home Credit - Credit Risk Model Stability
\n Create a model measured against feature stability over time\n","metadata":{}},{"cell_type":"markdown","source":"We have 32 files in train directory: **25 GB** in size in `csv` format or **1.3 GB** in `parquet` format. The choice is obvious - let's use `parquet`!\n\nWe have **450+ features** divided into six custom types. When you have almost half a thousand features from disparate sources, you shouldn’t expect anything good ;)\n\nI called this part of the EDA the first because there will be a second (of course if there are a lot of likes;)! And the second one will be, because the first one turned out to be quite large, even after analyzing only missing and zero values. \n\nOk! In this part I checks tables for missing values, zero values, and outliers. Why zero values too? Details below!\n\nAnd one more thing... It takes many pandas to defeat one polar bear...","metadata":{}},{"cell_type":"code","source":"# installation via pip is lightning fast, via conda - about 4 minutes\n# but when using pip there are problems (and sometimes not) with displaying hvplots \n\n! pip install /kaggle/input/install-pack-homecredit/hvplot-0.9.2-py2.py3-none-any.whl --no-deps\n#!conda install hvplot -y -q","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-03-03T04:24:24.220530Z","iopub.execute_input":"2024-03-03T04:24:24.220956Z","iopub.status.idle":"2024-03-03T04:24:28.022333Z","shell.execute_reply.started":"2024-03-03T04:24:24.220920Z","shell.execute_reply":"2024-03-03T04:24:28.021050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import polars as pl\nimport numpy as np\nimport pandas as pd\nfrom os.path import getsize, join, split, splitext\nfrom glob import glob\nfrom tqdm.notebook import tqdm\nfrom bokeh.models import NumeralTickFormatter\n\npl.Config(\n fmt_str_lengths=80,\n tbl_rows=80,\n set_thousands_separator=' ',\n float_precision=1,\n set_fmt_float=\"full\",\n tbl_cell_alignment = \"LEFT\",\n tbl_cell_numeric_alignment=\"RIGHT\"\n)\nfrmt_big_numb = NumeralTickFormatter(format='0.0a')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:24:28.024988Z","iopub.execute_input":"2024-03-03T04:24:28.025630Z","iopub.status.idle":"2024-03-03T04:24:29.551099Z","shell.execute_reply.started":"2024-03-03T04:24:28.025563Z","shell.execute_reply":"2024-03-03T04:24:29.549794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MAIN_PATH = '/kaggle/input/home-credit-credit-risk-model-stability'\nTRAIN_PATH_PARQUET = MAIN_PATH + '/parquet_files/train'\nTEST_PATH_PARQUET = MAIN_PATH + '/parquet_files/test'\nTRAIN_PATH_CSV = MAIN_PATH + '/csv_files/train'\nTEST_PATH_CSV = MAIN_PATH + '/csv_files/test'","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:24:29.552547Z","iopub.execute_input":"2024-03-03T04:24:29.553026Z","iopub.status.idle":"2024-03-03T04:24:29.558706Z","shell.execute_reply.started":"2024-03-03T04:24:29.552994Z","shell.execute_reply":"2024-03-03T04:24:29.557266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## General review","metadata":{}},{"cell_type":"markdown","source":"Here we examine our dataset for completeness. Are there many null values in the tables?","metadata":{}},{"cell_type":"code","source":"def aggregate_nulls(path):\n \n id = splitext(split(path)[-1])[0]\n feat_def = pl.read_csv(MAIN_PATH + '/feature_definitions.csv')\n \n df = pl.scan_parquet(path)\n df_dtypes = pl.DataFrame({'features':df.columns, 'dtypes':[str(x) for x in df.dtypes]})\n df = df.collect()\n \n df_shapes = pl.DataFrame({'file':id, 'n':df.shape[0], 'm':df.shape[1]}) \\\n .with_columns((pl.col('n')*pl.col('m')).alias('cells_numb'))\n \n df = df.null_count().select(pl.all().exclude(['case_id', 'num_group1', 'num_group2']), \n pl.lit(id).alias('file'),\n )\n \n df_nulls = df.melt('file', variable_name='features', value_name='nulls_numb') \\\n .with_columns(pl.col('features').str.slice(-1).alias('type')) \\\n .join(df_dtypes, on='features', how='left') \\\n .join(feat_def, left_on='features', right_on='Variable', how='left')\n \n return df_nulls, df_shapes","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:24:29.561455Z","iopub.execute_input":"2024-03-03T04:24:29.561971Z","iopub.status.idle":"2024-03-03T04:24:29.577983Z","shell.execute_reply.started":"2024-03-03T04:24:29.561938Z","shell.execute_reply":"2024-03-03T04:24:29.576541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# taking all files except 'train_base'\npath_list = [path for path in glob(f'{TRAIN_PATH_PARQUET}/*.parquet') if 'train_base.parquet' not in path]\n\nfor i, path in tqdm(enumerate(path_list), total=(len(path_list))):\n \n nulls, shapes = aggregate_nulls(path)\n \n if i==0:\n df_nulls, df_shapes = nulls, shapes\n else:\n df_nulls = df_nulls.vstack(nulls)\n df_shapes = df_shapes.vstack(shapes)\n \ndel nulls, shapes","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:24:29.579547Z","iopub.execute_input":"2024-03-03T04:24:29.580410Z","iopub.status.idle":"2024-03-03T04:25:26.631712Z","shell.execute_reply.started":"2024-03-03T04:24:29.580372Z","shell.execute_reply":"2024-03-03T04:25:26.630446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"null_prc = df_nulls.select(['file', 'nulls_numb']).group_by('file').sum() \\\n .join(df_shapes.select(['file','cells_numb']).group_by('file').sum(), on='file') \\\n .with_columns((pl.col('nulls_numb')/pl.col('cells_numb')).alias('null_prc')) \\\n .with_columns((1-pl.col('null_prc')).alias('filled_prc'))\n\ntot_nulls = (null_prc.select(\"nulls_numb\").sum()/null_prc.select(\"cells_numb\").sum())['nulls_numb'][0]\nprint(f'total number of null cells - {tot_nulls:.1%}, hense filled cells - {1-tot_nulls:.1%}')\n\ndisplay(\n null_prc.plot.bar(x='file',\n y=['filled_prc', 'null_prc'],\n stacked=True,\n rot=90,\n height=500,\n width=1000,\n title='Percentage of filled cells per table',\n ylabel='percentage of filled cells'\n )\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:25:26.633450Z","iopub.execute_input":"2024-03-03T04:25:26.633911Z","iopub.status.idle":"2024-03-03T04:25:30.863790Z","shell.execute_reply.started":"2024-03-03T04:25:26.633868Z","shell.execute_reply":"2024-03-03T04:25:30.862419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fell free to change threshold - it's funny!\nthreshold_of_nulls = 0.9 \n\nnulls_prc_per_feat = df_nulls.join(df_shapes, on='file').with_columns((pl.col('nulls_numb')/pl.col('n')).alias('nulls_prc'))\nft_wo_nulls = nulls_prc_per_feat.filter(pl.col('nulls_numb')==0) \\\n .group_by('file').agg(pl.col('features').count().alias('without nulls'))\nnull_above_th = nulls_prc_per_feat.filter(pl.col('nulls_prc')>threshold_of_nulls) \\\n .group_by('file').agg(pl.col('features').count().alias(f'more {threshold_of_nulls:.0%} nulls'))\nnull_below_th = nulls_prc_per_feat.filter((pl.col('nulls_prc')0)) \\\n .group_by('file').agg(pl.col('features').count().alias(f'less {threshold_of_nulls:.0%} nulls'))\n\ndf_pl = df_shapes.join(ft_wo_nulls, on='file', how='left') \\\n .join(null_above_th, on='file', how='left') \\\n .join(null_below_th, on='file', how='left') \\\n .fill_null(strategy='zero').sort('m', descending=True)\n\ndisplay(\n df_pl.plot.barh(x='file', \n y=['without nulls', f'less {threshold_of_nulls:.0%} nulls', f'more {threshold_of_nulls:.0%} nulls'], \n stacked=True, height=700, width=1200, legend='top_right', \n title = f'Number of features without nulls and with nulls in less/more then {threshold_of_nulls:.0%} of cases',\n ylabel='number of features',\n )\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:25:30.865469Z","iopub.execute_input":"2024-03-03T04:25:30.865966Z","iopub.status.idle":"2024-03-03T04:25:31.177988Z","shell.execute_reply.started":"2024-03-03T04:25:30.865932Z","shell.execute_reply":"2024-03-03T04:25:31.176723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nulls_prc_per_feat.select('nulls_prc').plot.hist(\n bins=10, \n title='Number of features by percentage of null values',\n xlabel='percentage of null values',\n ylabel='number of features'\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:25:31.179161Z","iopub.execute_input":"2024-03-03T04:25:31.179486Z","iopub.status.idle":"2024-03-03T04:25:31.977017Z","shell.execute_reply.started":"2024-03-03T04:25:31.179458Z","shell.execute_reply":"2024-03-03T04:25:31.975690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we've learned that our **dataset is 40% empty**. And it's not over yet!","metadata":{}},{"cell_type":"markdown","source":"## Table exploring","metadata":{}},{"cell_type":"markdown","source":"Now let’s examine individual tables in terms of features","metadata":{}},{"cell_type":"code","source":"\"\"\"\nHere we need to talk about one observation. When you select a value from the drop-down list, the chart may become empty. \nThis problem is solved by selecting the last one from a group of tables, after which you can select any of this group without problems. \nBy group I mean parts of divided tables. For example, if you select the table train_credit_buerau_a_0_1, the chart will be empty. \nFor the data to appear, you need to select the last one from the list of tables train_credit_buerau_a_0_*, namely train_credit_buerau_a_0_3. \nAfter this, you may return to the train_credit_buerau_a_0_1 table, and the data will be displayed correctly. Such a small unpleasant miracle!\n\nAnd one more thing! By default, the widget is located on the right in the middle, which is not very convenient, \nsince the chart is forced to be long. But after moving the widget to the top, the data stops updating altogether!\n\nThere is neither time nor competence to solve such problems. Maybe someone can...\n\"\"\"\nnulls_prc_per_feat = nulls_prc_per_feat.with_columns((1-pl.col('nulls_prc')).alias('filled_prc'))\ndisplay(\n nulls_prc_per_feat.plot.barh('features', 'filled_prc', by='type',\n groupby=['file'],\n stacked=True, \n dynamic=False,\n legend='bottom_right', \n# widget_location='right_top', # breakes down...\n height=1200, width=900,\n fontsize={'title': 12, 'xticks': 8, 'yticks': 8},\n hover_cols=['Description', 'dtypes'],\n ylabel='features filling percentage',\n\n )\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:25:31.978940Z","iopub.execute_input":"2024-03-03T04:25:31.979607Z","iopub.status.idle":"2024-03-03T04:25:35.869237Z","shell.execute_reply.started":"2024-03-03T04:25:31.979564Z","shell.execute_reply":"2024-03-03T04:25:35.868007Z"},"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The competition sponsor divided the features into custom types:\n\n> * **P** - Transform DPD (Days past due)\n> * **M** - Masking categories\n> * **A** - Transform amount\n> * **D** - Transform date\n> * **T** - Unspecified Transform\n> * **L** - Unspecified Transform\n\nLet's see how they intersect with built-in dtypes","metadata":{}},{"cell_type":"code","source":"# first of all explore intersection of the custom types and dtypes \ndf_nulls.pivot(index='type', columns='dtypes', values='nulls_numb', aggregate_function='len')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:25:35.874768Z","iopub.execute_input":"2024-03-03T04:25:35.875611Z","iopub.status.idle":"2024-03-03T04:25:35.898455Z","shell.execute_reply.started":"2024-03-03T04:25:35.875548Z","shell.execute_reply":"2024-03-03T04:25:35.897168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, custom types T and L, called *unspecified*, have a one-to-many relationship with built-in types.","metadata":{}},{"cell_type":"markdown","source":"Here is some functions that specifying plotting","metadata":{}},{"cell_type":"code","source":"# some functions that specifying plotting\n\ndef numb_rate_of_unique_table(df, feature, width, height, title):\n \"\"\"\n Create table with the number and percentage of unique values for categorical features\n \"\"\"\n table = df.select(pl.col(feature), pl.col(feature).count().alias('total')) \\\n .group_by([feature, 'total']).len().sort('len', descending=True) \\\n .filter(pl.col(feature).is_not_null()) \\\n .select(pl.col(feature), pl.col('len'),((pl.col('len')/pl.col('total'))*100).round(2).alias('rate')).collect() \\\n .plot.table(width=width, \n height=height,\n title=title,\n fontsize={'title':8},\n yformatter=frmt_big_numb)\n return table\n\ndef subplot_format(feat_list):\n \"\"\"\n Sets the number of columns and their height and width on a plot figure\n (these settings are specific to my monitor)\n \"\"\"\n len_feat_list = len(feat_list)\n if len_feat_list == 1:\n cols = 1\n width = 1000\n height = 300\n elif len_feat_list < 7:\n cols = 2\n width = 600\n height = 300\n else: \n cols = 3\n width = 400\n height = 250\n \n return cols, width, height\n\ndef plot_cat_features(df, feat_list):\n \"\"\"\n Plots data for categorical features\n \"\"\"\n cols, width, height = subplot_format(feat_list)\n \n # prepare a table for each categorical feature\n for i, feat in enumerate(feat_list):\n desc = take_description(feat)\n if not i:\n grid = numb_rate_of_unique_table(df, feat, width, height, desc)\n else:\n grid += numb_rate_of_unique_table(df, feat, width, height, desc)\n\n # display a graph of unique values per feature\n display(df.select(pl.col(feat_list).n_unique()).collect() \n .plot.barh(width=1000, \n xlabel='features', \n ylabel='number or unique values',\n title=f'Number of unique values per categorical feature; file - {file_name}'\n )\n )\n\n # display a tables of unique values with number and rate of each value\n display(grid.cols(cols)) if cols>1 else display(grid)\n \ndef plot_hist(df, feat_list, **ext_plot_options):\n \"\"\"\n Plots histogramms for non-categorical fratures\n \"\"\"\n df = df.collect()\n cols, width, height = subplot_format(feat_list)\n \n for i, feat in enumerate(feat_list):\n \n xlabel = take_description(feat)\n \n if not i:\n grid = df.plot.hist(feat, \n width=width, \n height=height,\n title=feat,\n xlabel = xlabel,\n fontsize={'xlabel':8},\n **ext_plot_options,\n )\n else:\n grid += df.plot.hist(feat, \n width=width, \n height=height,\n title=feat,\n xlabel = xlabel,\n fontsize={'xlabel':8},\n **ext_plot_options,\n )\n display(grid.cols(cols)) if cols>1 else display(grid)\n \ndef take_description(feat):\n \"\"\"\n Splits feature description into several lines of a given length\n \"\"\"\n maxlen = 70 # max length of feature descriptor line\n # take a feature description\n xlabel = df_nulls.filter(pl.col('features') == feat).select('Description').unique().item()[:-1]\n # there are many long feature descriptors, we should split them into several lines\n if len(xlabel) - maxlen > 10: # to avoid hyphenation of one-two letters\n xlabel = ''.join([xlabel[x:x+maxlen]+'\\n' for x in range(0, len(xlabel), maxlen)])[:-1]\n return xlabel\n\ndef describe_features(df, percentiles=(0.25,0.5,0.75)):\n \"\"\"\n Standard statistical description supplemented with:\n zero_count - number of zero values\n null% - percentage of null values\n null&zero - percentage of null and zero values together\n zero_in_not_null% - percentage of non-zero values in not empty(null) values\n \n \"\"\"\n df = df.collect()\n \n # get standart describe and add number of zero value\n df = pl.concat([df.describe(percentiles=percentiles),\n (df/df).select(pl.all().is_nan()).sum().cast(pl.Float64)\n .select(pl.lit('zero_count').alias('describe'), pl.all())]\n )\n \n names = df.select('describe').to_series()\n df = df.select(pl.exclude('describe')).transpose(include_header=True, header_name='feature', column_names=names)\n df = df.with_columns(((pl.col('null_count')/(pl.col('count')+pl.col('null_count')))*100\n ).alias('null%'),\n (\n (\n (pl.col('null_count')+pl.col('zero_count'))/\n (pl.col('count')+pl.col('null_count'))\n )*100\n ).alias('null&zero%'),\n ((pl.col('zero_count')/pl.col('count'))*100\n ).alias('zero_in_not_null%'),\n )\n return df","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:25:35.900400Z","iopub.execute_input":"2024-03-03T04:25:35.901223Z","iopub.status.idle":"2024-03-03T04:25:35.932452Z","shell.execute_reply.started":"2024-03-03T04:25:35.901172Z","shell.execute_reply":"2024-03-03T04:25:35.930797Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's specify the file to analyze and run everything below","metadata":{}},{"cell_type":"code","source":"###########################################################\n# HERE YOU SPECIFY THE FILE TO BE ANALYZED\n# feel free to paste here any tables (parquet format only!)\nfile_path = TRAIN_PATH_PARQUET + '/train_credit_bureau_a_1_1.parquet'\n###########################################################\nfile_name = splitext(split(file_path)[-1])[0]\ndf_test = pl.scan_parquet(file_path)","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:26:41.502587Z","iopub.execute_input":"2024-03-03T04:26:41.503044Z","iopub.status.idle":"2024-03-03T04:26:41.512900Z","shell.execute_reply.started":"2024-03-03T04:26:41.503014Z","shell.execute_reply":"2024-03-03T04:26:41.511119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Catigorical features (type M)","metadata":{}},{"cell_type":"markdown","source":"These features have only `string` types. To deside how to deal with them, let's look at how many unique values each has. Pay attention to cases of overwhelming predominance of one of the values","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*M$')).columns\nplot_cat_features(df_test, feat_list) if feat_list else print('There are not M-features')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:26:41.517827Z","iopub.execute_input":"2024-03-03T04:26:41.518391Z","iopub.status.idle":"2024-03-03T04:26:50.909217Z","shell.execute_reply.started":"2024-03-03T04:26:41.518342Z","shell.execute_reply":"2024-03-03T04:26:50.907725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Temporal features (type D)","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*D$')).columns\n\nif feat_list:\n plot_hist(df_test.select(pl.col(feat_list)).cast(pl.Date), \n feat_list,\n yformatter=frmt_big_numb,\n )\nelse:\n print('There are not D-features')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:26:50.912021Z","iopub.execute_input":"2024-03-03T04:26:50.912896Z","iopub.status.idle":"2024-03-03T04:27:03.578738Z","shell.execute_reply.started":"2024-03-03T04:26:50.912845Z","shell.execute_reply":"2024-03-03T04:27:03.577727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Amount features (type A)","metadata":{}},{"cell_type":"markdown","source":"The appearance of these plots is depressing (see below). A skyscraper in the area of zero, and desert for many miles... This \"landscape\" led me to the idea of the need for a quantitative analysis of zero values. It turns out that up to 100 percent of non-empty data can simply be zeros! These are, of course, only parts of the divided tables... In the second part of the study, we will check the situation in the joined tables","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*A$')).columns\n\nif feat_list:\n display(describe_features(df_test.select(pl.col(feat_list))))\n plot_hist(df_test.select(pl.col(feat_list)), \n feat_list, \n xformatter=frmt_big_numb, \n yformatter=frmt_big_numb,\n ) \nelse:\n print('There are not A-features')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:03.580161Z","iopub.execute_input":"2024-03-03T04:27:03.580827Z","iopub.status.idle":"2024-03-03T04:27:24.153514Z","shell.execute_reply.started":"2024-03-03T04:27:03.580787Z","shell.execute_reply":"2024-03-03T04:27:24.152119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To at least see something, let's cut off what is beyond two sigma (to eliminate outliers) and logarithm the result...\n\nWow! It's much better! Now we have a whole city of skyscrapers of different heights. With this you can already test the hypothesis of a normal distribution!\n\nTo be honest, there's a little trick here... Not every feature will behave so well. I chose it based on the presence of values other than zero in all quartiles. It's like the birth of an algorithm... Anyway, it's food for thought!","metadata":{}},{"cell_type":"code","source":"# for example let's take 'outstandingamount_362A' from train_credit_bureau_a_1_1.parquet\n# if you are analyzing another table there will be a random feature\nif feat_list:\n if 'outstandingamount_362A' in feat_list:\n feat = 'outstandingamount_362A'\n else:\n feat = feat_list[np.random.randint(0, len(feat_list))]\n dd_plot = df_test.select(feat).collect() \\\n .filter(pl.col(feat).is_between(pl.col(feat).quantile(0.02), pl.col(feat).quantile(0.98))) \\\n .select((pl.col(feat)+1).log()) \n display(dd_plot.plot.hist(width=600) + dd_plot.plot.kde(filled=False, width=600))","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:24.157096Z","iopub.execute_input":"2024-03-03T04:27:24.157523Z","iopub.status.idle":"2024-03-03T04:27:26.378386Z","shell.execute_reply.started":"2024-03-03T04:27:24.157487Z","shell.execute_reply":"2024-03-03T04:27:26.377140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### DPD (Days past due) features (type P)","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*P$')).columns\n\nif feat_list:\n display(describe_features(df_test.select(pl.col(feat_list))))\n plot_hist(df_test.select(pl.col(feat_list)), \n feat_list, \n xformatter=frmt_big_numb, \n yformatter=frmt_big_numb,\n ) \nelse:\n print('There are not P-features')\n","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:26.380238Z","iopub.execute_input":"2024-03-03T04:27:26.380620Z","iopub.status.idle":"2024-03-03T04:27:28.367744Z","shell.execute_reply.started":"2024-03-03T04:27:26.380587Z","shell.execute_reply":"2024-03-03T04:27:28.366331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Features with unspecified transform (type L)","metadata":{}},{"cell_type":"markdown","source":"*L-features* may be three dtypes: `string`, `float`, and `boolean`","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*L$')).select(pl.col(pl.Float64)).columns\n\nif feat_list:\n display(describe_features(df_test.select(pl.col(feat_list))))\n plot_hist(df_test.select(pl.col(feat_list)), \n feat_list, \n xformatter=frmt_big_numb, \n yformatter=frmt_big_numb,\n ) \nelse:\n print('There are not L-features with type \"float64\"')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:28.369193Z","iopub.execute_input":"2024-03-03T04:27:28.370279Z","iopub.status.idle":"2024-03-03T04:27:44.666245Z","shell.execute_reply.started":"2024-03-03T04:27:28.370241Z","shell.execute_reply":"2024-03-03T04:27:44.664581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*L$')).select(pl.col(pl.Boolean)).columns\n\nif feat_list:\n plot_hist(df_test.select(pl.col(feat_list)).cast(pl.UInt8), \n feat_list, \n bins=2,\n yformatter=frmt_big_numb,\n ) \nelse:\n print('There are not L-features with type \"boolean\"')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:44.668093Z","iopub.execute_input":"2024-03-03T04:27:44.668464Z","iopub.status.idle":"2024-03-03T04:27:44.677185Z","shell.execute_reply.started":"2024-03-03T04:27:44.668435Z","shell.execute_reply":"2024-03-03T04:27:44.675850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*L$')).select(pl.col(pl.String)).columns\nplot_cat_features(df_test, feat_list) if feat_list else print('There are not L-features with type \"string\"')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:44.678802Z","iopub.execute_input":"2024-03-03T04:27:44.679252Z","iopub.status.idle":"2024-03-03T04:27:44.695022Z","shell.execute_reply.started":"2024-03-03T04:27:44.679220Z","shell.execute_reply":"2024-03-03T04:27:44.693625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Features with unspecified transform (type L)","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*T$')).select(pl.col(pl.String)).columns\nplot_cat_features(df_test, feat_list) if feat_list else print('There are not T-features with type \"string\"')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:44.696291Z","iopub.execute_input":"2024-03-03T04:27:44.696637Z","iopub.status.idle":"2024-03-03T04:27:44.711032Z","shell.execute_reply.started":"2024-03-03T04:27:44.696609Z","shell.execute_reply":"2024-03-03T04:27:44.709639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this particular case, we find that numeric features of type T contain the index numbers of the year or month","metadata":{}},{"cell_type":"code","source":"feat_list = df_test.select(pl.col(r'^.*T$')).select(pl.col(pl.Float64)).columns\n\nif feat_list:\n display(describe_features(df_test.select(pl.col(feat_list))))\n plot_hist(df_test.select(pl.col(feat_list)), \n feat_list, \n xformatter=frmt_big_numb, \n yformatter=frmt_big_numb,\n ) \nelse:\n print('There are not T-features with type \"float64\"')","metadata":{"execution":{"iopub.status.busy":"2024-03-03T04:27:44.714371Z","iopub.execute_input":"2024-03-03T04:27:44.714822Z","iopub.status.idle":"2024-03-03T04:27:52.167041Z","shell.execute_reply.started":"2024-03-03T04:27:44.714788Z","shell.execute_reply":"2024-03-03T04:27:52.165517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"TODO:\n - Analise joined tables\n - Determine the criteria for selecting features\n - Selection of aggregation functions, analysis of grouped data\n - Some statistical research\n - Something else...","metadata":{}}]}
![](\"https://www.kaggle.com/competitions/50160/images/header\")
\n Home Credit - Credit Risk Model Stability