{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n

This is an task to predict whether the APP will be installed (is_attributed=True) after user click the AD. The data-set is un-balanced (`positive/negative = 0.03/1`) and is too big for single-server environment. Belows are the challenge and solutions during the model training.

\n\n* **Memore Insufficient when Data Loading**: the data-set is too large to load into memory, thus the data is partioned before feature-extraction and down-sampling. And then the down-sampled data are merged together as the training set
\n* **Model Evaluation**: If fill down-sampled data into the verification-set, it can not reflect the real performace in production. Thus the original data, i.e non-down-sampled data, is used as the verification set in parameter search and model evaluation. Also a function is writen to support this kind of parameter search\n* **Parameter Change Trace**: Simple common code is implemented to trace base-line models and experiments (parameter-serch) base on these baselines.\n* **Feature Selections**: basically is based on the feature-importency. Since the testing-set is hour-based (only several hour data), the day-based feature will be in-consistent between training set and testing set. Thus, the features are only aggregated by hour or by all-data, not by day. \n\n> 任务目标是在一个单机环境难以承载的不平衡数据集（正例/负例 = 0.03/1）上预测一次广告点击（AdClick）是否可以带来APP下载安装。以下是所面临的挑战和解决办法：
\n> \n> * **数据集载入时的内存问题**：该数据集样本数量充足，但无法载入内存，为了充分使用稀缺的正例样本，对训练集采用了“文件分桶 -> 特征提取 -> 降采样 -> 合并”的处理方式\n> * **不平衡数据集模型评估**：降采样数据作为验证集的分数（虽然通常非常好）不能反映生产环境的真实效果，为了正确评估生产环境的模型效果，在搜索模型参数、评估模型时，都使用未经降采样的原始数据作为验证集。编写了简单的搜索模型参数的代码，来实现在参数搜索中有选择性地降采样。\n> * **实验参数管理**：吸取前一篇[Note](https://www.kaggle.com/fangkun119/competition-mercedes-benz-greener-manufacturing)的教训，编写了简单的模型参数管理代码，用来追溯每个实验（参数搜索）以及对应的基线模型，让调参过程有迹可循\n> * **特征选择**：基于lightgbm内部决策树给出的特征重要度、以及<特征-特征>、<特征-目标>相关度来选择特征等。由于竞赛给出的测试集是三个小时片段，使用天级别的特征会导致训练数据与测试数据不一致，因此剔除天级特征，只使用了多日汇总特征以及小时级特征，并让模型来决定更加倾向于哪一种特征\n"},{"metadata":{},"cell_type":"markdown","source":"# libraries and Util Functions"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# python util\nimport os\nimport gc\nfrom operator import methodcaller\n\n# data processing\nimport numpy as np\nimport pandas as pd\nfrom sklearn.preprocessing import RobustScaler as RobustScaler\nfrom scipy.stats import skew, norm\nfrom scipy.stats import boxcox_normmax, boxcox\nfrom scipy.special import boxcox1p\n\n# lightgbm\nimport lightgbm as lgb\nfrom lightgbm import LGBMClassifier\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.model_selection import GridSearchCV\n\n# plot\nimport matplotlib.pyplot as plt \nimport seaborn as sns \n\n# random seed\n# import random\n# random.seed(567)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Util funtions \n\n```python\ng_enable_log = True\ndef log(log_str)\ndef g() #short-cut of gc.collect()\ndef delete(*obj_list)\ndef robust_boxcox(data:pd.Series, lmbda=None)\ndef robust_inv_boxcox(data:pd.Series, lmbda)\n@contextmanager\ndef timer_memory(name)\n```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"g_enable_log = True\ndef log(log_str):\n if g_enable_log:\n print(log_str)\n\ndef g():\n return gc.collect()\n\ndef delete(*obj_list):\n for obj in obj_list:\n del obj\n gc_cnt = g()\n if gc_cnt > 0:\n log(\"unreachable_obj_found: {}\".format(gc_cnt))\n\n#reference:\n#https://stackoverflow.com/questions/47213443/divide-by-zero-encountered-in-log-scipy-stats-boxcox\ndef init_robust_boxcox():\n from scipy.special import inv_boxcox\n from scipy.stats import boxcox\n\n def robust_boxcox(data:pd.Series, lmbda=None):\n if lmbda is None:\n transformed, lmbda = boxcox(1 + data)\n # enhance in condition that boxcox returns \n # a extremely close-to-zero negative float instead of return 0\n if lmbda <= 0:\n lmbda = 0\n transformed = np.log1p(data) #log(1+data)\n return (transformed, lmbda)\n else:\n if lmbda <= 0:\n lmbda = 0\n transformed = np.log1p(data) #log(1+data)\n return (transformed, lmbda)\n else:\n transformed = boxcox(1 + data, lmbda)\n return (transformed, lmbda)\n\n def robust_inv_boxcox(data:pd.Series, lmbda):\n if lmbda <= 0:\n return np.expm1(data) #inv of log1p\n else:\n transformed = inv_boxcox(data, lmbda) - 1\n return transformed\n \n return robust_boxcox, robust_inv_boxcox\n\nrobust_boxcox, robust_inv_boxcox = init_robust_boxcox()\n\nimport time\nimport psutil\nfrom contextlib import contextmanager \n@contextmanager\ndef timer_memory(name):\n t0 = time.time()\n yield\n print(f'Memory: {(psutil.Process(os.getpid()).memory_info().rss/2**30):.02f}GB')\n print(f'{name} done in {time.time()-t0:.0f}s')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Input Files\n\n

Input Files

\n
  • train.csv: training set, the 1st column is IP, records are sorted by click_time
    • \n
  • test.csv: testing set, the 1st column is click_id (from 0 to record_num-1), 2nd column is IP, sorted by click_time
    • \n

    \n\n> 项目文件：\n>
  • train.csv: 训练集数据, 第1列是IP，按click_time排序
    • \n>
  • test.csv: 测试集数据，第1列是click_id，第2列是IP
    • "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n!head -n3 /kaggle/input/talkingdata-adtracking-fraud-detection/train.csv\n!head -n3 /kaggle/input/talkingdata-adtracking-fraud-detection/test.csv ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

    Data Partitioning

    \n1. split the dataset into 20 partitions according to the value of IP % 20, which is named as IPBucket\n2. each bucket is smaller enough to be loaded into memory, now we can add whatever features we want to try without worrying about the memory usage\n3. down-sample the training set to balance the data-set. Since majory(is_attributed=1)/minority(is_attributed=0) is 99.7/0.3, down-sample will reduce training set to 0.3% as before as the same time\n4. fit model with down-sampled training-set (the verification-set and testing-set won't be down-sampled)\n \n> 数据分桶\n> * 按照IP%20将数据集分成20个桶，提取特征；随后对训练集进行降采样、使得样本标签(is_attributed)分布均衡，同时也降低了训练集的数据量\n> * 用将采样的训练集、以及保持原始分布(未经过将采样)的验证集训练模型；测试集则用于提交最终预测结果
    \n> 以下是数据分桶的代码"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\n# create directories and make sure they are empty\n!cd /kaggle/input/\n!mkdir -p /kaggle/input/train /kaggle/input/test \n!rm -f /kaggle/input/train/* /kaggle/input/test/*\n\n# full data with 20 bucket \n# write column names into each file\n!for i in $(seq 0 19); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' > /kaggle/input/train/train_${i}.csv; done\n!for i in $(seq 0 19); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' > /kaggle/input/test/test_${i}.csv; done\n# split the data set according to IP(1st column of train.csv, 2nd column of test.csv) % 20\n!cat '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' | grep -v '^[a-zA-Z]' | awk -F\",\" '{print $0 >> \"/kaggle/input/train/train_\"$1%20\".csv\"}'\n!cat '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' | grep -v '^[a-zA-Z]' | awk -F\",\" '{print $0 >> \"/kaggle/input/test/test_\"$2%20\".csv\"}'\n# configurations for next steps\ng_ip_bkt_num = 20 # partition number\ng_vldt_set_size = 5000000 # validation set size\ng_is_down_sample = True # configration to enable the tain-set down-sample\ng_majority_multiply = 1 # configration to decide majority record numbers after down sample \n # 1: means majority_record_number = minority_record_number\n # 2: means majority_record_number = minority_record_number * 2\ng_scale_pos_weight = 1 # positive example weight (use to set scale_pos_weight parameter of lightgbm)\n # integer: positive example weight = g_s negative example\n # none: won't set scale_pos_weight, \n # but set is_unbalanced=True to let lightgbm decide the weight by itself\n# check data\n!head -n3 /kaggle/input/train/* /kaggle/input/test/* # first 3 lines of each file\n!ls -lh /kaggle/input/train/* /kaggle/input/test/* # file size\n!wc -l /kaggle/input/train/* /kaggle/input/test/* # record numbers of each file","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If don't want to partition these data, or just want to use a small dataset to debug the program, we can un-comment the code below."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#%%time\n#!cd /kaggle/input/\n#!mkdir -p /kaggle/input/train /kaggle/input/test \n#!rm -f /kaggle/input/train/* /kaggle/input/test/*\n\n#debug with 45000 data with 1 bucket\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' > /kaggle/input/train/train_${i}.csv; done\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' > /kaggle/input/test/test_${i}.csv; done\n#!tail -n 45000 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' | grep -v '^[a-zA-Z]' | awk -F\",\" '{print $0 >> \"/kaggle/input/train/train_\"$1%1\".csv\"}'\n#!cat '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' | grep -v '^[a-zA-Z]' |awk -F\",\" '{print $0 >> \"/kaggle/input/test/test_\"$2%1\".csv\"}'\n#g_ip_bkt_num = 1\n#g_vldt_set_size = 50 \n#g_is_down_sample = True\n#g_scale_pos_weight = 1\n#g_majority_multiply = 1\n\n#latest 30000000 data with 1 bucket\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' > /kaggle/input/train/train_${i}.csv; done\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' > /kaggle/input/test/test_${i}.csv; done\n#!tail -n 30000000 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' | grep -v '^[a-zA-Z]' >> \"/kaggle/input/train/train_0.csv\" \n#!cat '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' | grep -v '^[a-zA-Z]' >> \"/kaggle/input/test/test_0.csv\"\n#g_ip_bkt_num = 1\n#g_vldt_set_size = 5000000\n#g_is_down_sample = False\n#g_scale_pos_weight = 99.7\n#g_majority_multiply = 1\n\n#check data\n#!head -n3 /kaggle/input/train/* /kaggle/input/test/*\n#!ls -lh /kaggle/input/train/* /kaggle/input/test/*\n#!wc -l /kaggle/input/train/* /kaggle/input/test/*","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering\n\n```python\n# feature_name -> scaler\ng_scaler_dict = {}\n# feature_name -> lmbda\ng_boxcox_lmbda_dict = {} \ndef box_cox_trans(df, fea_name, sv_policy)\ndef scaler_trans(df, fea_name, sv_policy, scaler)```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"g_scaler_dict = {} # feature_name -> scaler\ng_boxcox_lmbda_dict = {} # feature_name -> lmbda\n\ndef box_cox_trans(df, fea_name, sv_policy):\n '''sv_policy (lmbda saving policy): new_and_save, reuse, new'''\n df[fea_name] = df[fea_name].astype('float64'); g()\n df.loc[df[fea_name] <= 0, (fea_name)] = 0.000001\n if sv_policy == 'new_and_save':\n df[fea_name], lmbda = robust_boxcox(df[fea_name]); g()\n g_boxcox_lmbda_dict[fea_name] = lmbda\n elif sv_policy == 'reuse':\n lmbda = g_boxcox_lmbda_dict[fea_name]\n df[fea_name], lmbda = robust_boxcox(df[fea_name], lmbda); g()\n else:\n df[fea_name], lmbda = robust_boxcox(df[fea_name]); g()\n log(\"\\tboxcox lmbda: {}\".format(lmbda))\n return df[fea_name].astype('float32')\n\ndef scaler_trans(df, fea_name, sv_policy, scaler):\n '''sv_policy (scaler saving policy): new_and_save, reuse, new''' \n df[fea_name] = df[fea_name].astype('float64'); g()\n if sv_policy == 'new_and_save':\n df[[fea_name]] = scaler.fit_transform(df[[fea_name]]); g()\n g_scaler_dict[fea_name] = scaler\n elif sv_policy == 'reuse':\n df[[fea_name]] = g_scaler_dict[fea_name].transform(df[[fea_name]]); g()\n else: #'new'\n df[[fea_name]] = scaler.fit_transform(df[[fea_name]]); g()\n return df[fea_name].astype('float32') ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

    functions for creating new features

    \n> 用于提取特征的函数\n\n```python\ndef add_grp_count(df, group_columns, dtype='uint32', scl=None, bc=False, sv='reuse')\ndef add_grp_stat(df, group_columns, stat_column, stat_fun_name, dtype, scl=None, bc=False, sv='reuse')\ndef add_grp_nxt_clk_intv(df, group_columns, scl=None, bc=False, sv='reuse')```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# parameters of below functions:\n# scl : scaler object\n# bc : is_boxcox, whether enable boxcox transform\n# sv : how to get transformer when boxcox tranforming and scaling\n# 'new': create new transformer\n# 'new_and_save': create new transformer and save it for future\n# 'reuse': reuse existing transformer\n\ndef add_grp_count(df, group_columns, dtype='uint32', scl=None, bc=False, sv='reuse'):\n '''\n steps: \n 1. group DataFrame(df) by group_columns\n 2. get the group size\n 3. attach group size to each rows\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv \n # feature name\n feature_name = \"cnt_grp_by_\" + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n # group-count\n group_col_and_cnt = df.groupby(group_columns).size().astype(dtype); g()\n # format-conversion\n group_col_and_cnt = group_col_and_cnt.rename(feature_name).to_frame().reset_index(); g()\n # merge\n df_tmp = df.merge(group_col_and_cnt, on=group_columns, how='left'); delete(df)\n df = df_tmp\n # boxcox\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g()\n # scaler\n if scaler is not None: #1\n df[feature_name] = scaler_trans(df, feature_name, sv_policy=sv, scaler=scaler); g() \n # return\n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}')\n return df\n\ndef add_grp_stat(df, group_columns, stat_column, stat_fun_name, dtype, scl=None, bc=False, sv='reuse'):\n '''\n steps:\n 1. group DataFrame(df) by group_columns \n 2. apply stat_fun_name on stat_column in each group\n 3. attach stat_func_name's ouput to each rows\n stat_func_nane: \"cumcount\", \"var\", \"count\", \"nunique\"\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv \n # feature name\n feature_name = stat_fun_name + \"_on_\" + stat_column + \"_grp_by_\" + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n # group then stat in each group\n grouped = df[group_columns + [stat_column]].groupby(group_columns); g()\n stat_applied = methodcaller(stat_fun_name)(grouped[stat_column]).astype(dtype); g()\n delete(grouped)\n if stat_fun_name == \"var\":\n stat_applied = stat_applied.transform(lambda x: x.fillna(0)); g()\n # data frame conversion\n group_columns_and_stat_column = stat_applied.rename(feature_name).to_frame()\n delete(stat_applied)\n # merge\n if stat_fun_name == \"cumcount\": \n df[feature_name] = group_columns_and_stat_column[feature_name].astype(dtype); g()\n elif stat_fun_name in ['var', 'count', 'nunique']:\n group_columns_and_stat_column = group_columns_and_stat_column.astype(dtype); g()\n group_columns_and_stat_column = group_columns_and_stat_column.reset_index(); g()\n df_tmp = df.merge(group_columns_and_stat_column, on=group_columns, how='left')\n delete(df)\n df = df_tmp\n else:\n raise Exception('un-supported stat_fun_name: {}'.format(stat_fun_name)) \n # boxcox transform and scaler\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g()\n if scaler is not None: #2\n df[feature_name] = scaler_trans(df, feature_name, sv_policy=sv, scaler=scaler); g()\n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}') \n return df\n\ndef add_grp_nxt_clk_intv(df, group_columns, scl=None, bc=False, sv='reuse'):\n '''\n add column by grouping then calculate click interval \n between the record and it's next click in the same group\n steps:\n 1.group DataFrame(df) by group_columns \n 2.calculate next click_time in the same group for each given record\n 3.calculate intervial between current click and it's next click in the same group\n 4.merge the intervial time into the data set\n 5.scale the intervial time with RobustScaler if is_scale is True\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv\n # feature name\n feature_name = 'nxt_itvl_by_' + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n # interval to next click\n df['click_time_in_sec'] = (df['click_time'].astype(np.int64)//10**9).astype(np.int32)\n df[feature_name] = (df.groupby(group_columns)['click_time_in_sec'].shift(-1) - df['click_time_in_sec']).astype(np.float32); g()\n df[feature_name] = df[feature_name].fillna(df[feature_name].mean()); g()\n print('\\tfillna: {}'.format(df[feature_name].mean()))\n df.drop(['click_time_in_sec'], axis=1, inplace=True)\n # boxcox transform and scaler\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g()\n if scaler is not None: #3\n df[feature_name] = scaler_trans(\n df, feature_name, sv_policy=sv, scaler=scaler); g()\n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}') \n return df\n\ndef add_grp_pre_clk_intv(df, group_columns, scl=None, bc=False, sv='reuse'):\n '''\n add column by grouping then calculate click interval \n between the record and it's previouse click in the same group\n steps:\n 1.group DataFrame(df) by group_columns \n 2.calculate previous click_time in the same group for each given record\n 3.calculate the interval between click_time of the record \n and it's previouse click_time in the same group\n 4.merge the intervial time into the data set\n 5.scale the intervial time with RobustScaler if is_scale is True\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv \n # feature name\n feature_name = 'pre_itvl_by_' + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n # calculate click interval\n df['click_time_in_sec'] = (df['click_time'].astype(np.int64)//10**9).astype(np.int32)\n df[feature_name] = (df['click_time_in_sec'] - df.groupby(group_columns)['click_time_in_sec'].shift(1)).astype(np.float32); g()\n df[feature_name] = df[feature_name].fillna(df[feature_name].mean()); g()\n print('\\tfillna: {}'.format(df[feature_name].mean()))\n df.drop(['click_time_in_sec'], axis=1, inplace=True)\n # boxcox transform and scaler\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g() \n if scaler is not None: #4\n df[feature_name] = scaler_trans(df, feature_name, sv_policy=sv, scaler=scaler); g() \n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}') \n return df\n\n# reference\n# help(pandas._libs.tslibs.timedeltas.Timedelta)\n# help(pandas._libs.tslibs.nattype.NaTType)\n# https://kapeli.com/dash_share?docset_file=Pandas&docset_name=Pandas&path=doc/reference/series.html%23timedelta-properties&platform=pandas&repo=Main&version=0.25.1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

    extract features

    "},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_features(df, is_boxcox=False, is_scaler=False, save_transformer='reuse'):\n '''\n create extended features\n parameters:\n * save_transformer: how to get transformer when doing boxcox and scaling features\n * 'new': create new transformer\n * 'new_and_save': create new transformer and save it for future\n * 'reuse': reuse existing transformer\n '''\n # parameter short cut\n sv = save_transformer \n bc = is_boxcox\n s = None;\n if is_scaler:\n s = RobustScaler()\n # features and importancy\n # 2008: nxt_intv_by_ip_os_device_app\n df = add_grp_nxt_clk_intv(df, ['ip','os','device','app'], bc=None,scl=s,sv=sv);g()\n # 1747: channel\n # 1313: os\n # 1182: hh\n # 1051: app\n # 1035: cnt_grp_by_dd_hh_app_channel\n df = add_grp_count(df, ['dd','hh','app','channel'], bc=bc,scl=s,sv=sv);g()\n # 995 : cumcount_on_app_grp_by_ip_device_os\n df = add_grp_stat(df, ['ip','device','os'], 'app', 'cumcount', 'uint32', bc=bc,scl=s,sv=sv);g()\n # 990 : cnt_grp_by_ip_device\n df = add_grp_count(df, ['ip','device'], bc=bc,scl=s,sv=sv);g()\n # 929 : nunique_on_channel_by_ip\n df = add_grp_stat(df, ['ip'], 'channel', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g()\n # 921 : nunique_on_app_by_ip\n df = add_grp_stat(df, ['ip'], 'app', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g()\n # 835 : nxt_intv_by_ip_channel\n df = add_grp_nxt_clk_intv(df, ['ip','channel'], bc=None,scl=s,sv=sv);g()\n # 834 : cnt_grp_by_ip_hh_device\n df = add_grp_count(df, ['ip','hh','device'], bc=bc,scl=s,sv=sv);g()\n # 832 : nxt_intv_by_ip_app_channel\n df = add_grp_nxt_clk_intv(df, ['ip','app','channel'], bc=None,scl=s,sv=sv);g()\n # 825 : cnt_grp_by_app_channel\n df = add_grp_count(df, ['app','channel'], bc=bc,scl=s,sv=sv);g()\n # 716 : cnt_grp_by_ip_app\n df = add_grp_count(df, ['ip','app'], bc=bc,scl=s,sv=sv);g()\n # 635 : unique_on_app_grp_by_ip_hh\n df = add_grp_stat(df, ['ip','hh'], 'app', 'nunique','uint16',bc=bc,scl=s,sv=sv);g()\n # 592 : cnt_grp_by_ip_hh_app\n df = add_grp_count(df, ['ip','hh','app'], bc=bc,scl=s,sv=sv);g()\n # 432 : nunique_on_channel_by_app\n df = add_grp_stat(df, ['app'], 'channel', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g() \n # 422 : nunique_on_channel_by_hh_app\n df = add_grp_stat(df, ['hh','app'], 'channel', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g() \n # 307 : device\n return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

    function for random down-sampling, normal we setting majority_multiply=1 to make majority_examples_count/minority_examples_count=1, sometime we want more majority examples we can increase majority_multiply and vice versa

    \n> 用于随机降采样的函数，设置majority_multiply=1可以让majority样本数与minority样本数相同，增加majority_multiply可以增加majority样本数占比，反之亦然\n\n```python\ndef random_down_sample(df, majority_multiply=1, target_col_name='is_attributed', minority_val = 1, majority_val = 0)```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def random_down_sample(df, majority_multiply=1, target_col_name='is_attributed', minority_val=1, majority_val=0):\n '''\n down sample the majority part, \n so that both part (target=1, target=0) has equal number of recordes\n '''\n minority_cnt = len((df[df[target_col_name] == minority_val])) \n majority_cnt = (minority_cnt * majority_multiply) // 1\n\n shuffled = df.sample(frac=1)\n minority = shuffled.loc[shuffled[target_col_name] == minority_val]\n majority = shuffled.loc[shuffled[target_col_name] == majority_val][:majority_cnt]\n\n undersample_df = pd.concat([minority, majority]).sample(frac=1)\n delete(shuffled, minority, majority)\n return undersample_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare Data Set\n\n## configures"},{"metadata":{"trusted":true},"cell_type":"code","source":"# categorical features\ng_categorical_features = ['app', 'device', 'os', 'channel', 'hh'] \n\n# only for extracting features, won't used in model training\ng_non_train_columns = ['click_time', 'dd', 'ip']\n\n# file spec\ndef get_file_spec(is_test_file):\n dtypes = {'ip':'uint32','app':'uint16','device':'uint8','os':'uint16',\n 'channel':'uint16','is_attributed':'int8','click_id':'int32'} \n date_columns = ['click_time']\n test_file_columns = ['click_time','ip','app','device','os','channel','click_id']\n train_file_columns = ['click_time','ip','app','device','os','channel','is_attributed'] \n if is_test_file:\n return dtypes, date_columns, test_file_columns\n else:\n return dtypes, date_columns, train_file_columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## functions\n\n

    functions for preparing data set

    \n\n> 用于准备数据集的函数\n\n```python\ndef read_data_file(file_path, is_test_file)\ndef process_ip_bucket(ip_bucket, is_down_sample, majority_multiply, tsfm_sv_policy) #'new_and_save','reuse','new'\n\n# columns created in read_data_file(file_path, is_test_file)\ndf['dd'] = pd.to_datetime(df.click_time).dt.day.astype('uint8') # [01, 31]\ndf['hh'] = pd.to_datetime(df.click_time).dt.hour.astype('uint8') # [00, 23]```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def read_data_file(file_path, is_test_file):\n print('read file [is_test_file={}]: {}'.format(is_test_file, file_path))\n dtypes, date_columns, file_columns = get_file_spec(is_test_file)\n df = pd.read_csv(file_path, parse_dates=date_columns,usecols=file_columns,dtype=dtypes)\n df['dd'] = pd.to_datetime(df.click_time).dt.day.astype('uint8') # [01, 31]\n df['hh'] = pd.to_datetime(df.click_time).dt.hour.astype('uint8') # [00, 23]\n g()\n return df\n\ndef process_ip_bucket(ip_bucket, is_down_sample, majority_multiply, tsfm_sv_policy):\n # file path\n print('---------- process bucket: {} -----------'.format(ip_bucket))\n train_file_path = '/kaggle/input/train/train_{}.csv'.format(ip_bucket)\n test_file_path = '/kaggle/input/test/test_{}.csv'.format(ip_bucket)\n # read file\n df_test = read_data_file(test_file_path, is_test_file=True) \n df_train_vldt = read_data_file(train_file_path, is_test_file=False)\n print('read files: {}; {}'.format(train_file_path, test_file_path))\n # align columns and merge\n df_train_vldt['click_id'] = -1\n df_test['is_attributed'] = -1\n df_full = pd.concat([df_train_vldt, df_test], sort=False)\n # train_vldt_len and nunique of raw feature\n train_vldt_len = len(df_train_vldt)\n delete(df_train_vldt, df_test)\n #print('df_full.head(n=5000000).nunique():\\n', df_full.head(n=5000000).nunique()); g()\n #ip:9819; app:343; device:250; os:246; channel:169; click_time:121600; ...\n # feature process\n print('add features: ')\n df_full = add_features(df_full, save_transformer=tsfm_sv_policy); g()\n # drop non-training columns\n print('drop non-training-columns: {}'.format(g_non_train_columns))\n df_full.drop(g_non_train_columns, axis=1, inplace=True); g()\n # split into df_train, df_vldt, df_test\n print('split data set: ')\n vldt_len = min(g_vldt_set_size // g_ip_bkt_num, train_vldt_len // 5)\n df_train = df_full[: train_vldt_len - vldt_len]\n df_vldt = df_full[train_vldt_len - vldt_len : train_vldt_len]\n df_test = df_full[train_vldt_len :]\n # drop tmp colums created when aligning\n print('drop temporary columns: ')\n print('\\tdf_train[\\'click_id\\'].value_counts()={}'.format(df_train['click_id'].value_counts()))\n print('\\tdf_vldt[\\'click_id\\'].value_counts()={}'.format(df_vldt['click_id'].value_counts()))\n print('\\tdf_test[\\'is_attributed\\'].value_counts()={}'.format(df_test['is_attributed'].value_counts()))\n df_train.drop(['click_id'], axis=1, inplace=True)\n df_vldt.drop(['click_id'], axis=1, inplace=True)\n df_test.drop(['is_attributed'], axis=1, inplace=True)\n print('shape: vldt={}; test={}'.format(df_vldt.shape, df_test.shape))\n print('shape: train(before downsample)={}'.format(df_train.shape))\n g()\n # downsample if necessary\n if is_down_sample == True:\n print('down_sample: majority_multiply={}'.format(majority_multiply))\n df_train = random_down_sample(df_train, majority_multiply)\n print('gc.collect (warnning is OK): ')\n g()\n print('shape(after downsample)={}'.format(df_train.shape))\n # return\n return df_train, df_vldt, df_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare Data Set\n\n> 准备数据集\n\n```python\ndef prep_data_set_full_data()\ndef feature_target_split(df, target_col_name, inplace=False)\ndef prep_feature_target_full_data()\n```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def prep_data_set_full_data():\n log_template=\"append bkt {}: train.shape={}; vldt.shape={}; test.shape={}\"\n df_train, df_vldt, df_test = process_ip_bucket(\n ip_bucket=0, is_down_sample=g_is_down_sample, \n majority_multiply=g_majority_multiply, \n tsfm_sv_policy='new_and_save')\n print(log_template.format(0, df_train.shape, df_vldt.shape, df_test.shape))\n for bkt_id in range(1,g_ip_bkt_num):\n train_bkt, vldt_bkt, test_bkt = process_ip_bucket(\n ip_bucket=bkt_id, is_down_sample=g_is_down_sample, \n majority_multiply=g_majority_multiply, \n tsfm_sv_policy='reuse'); g()\n df_train = df_train.append(train_bkt, ignore_index=True); g()\n df_vldt = df_vldt.append(vldt_bkt, ignore_index=True); g()\n df_test = df_test.append(test_bkt, ignore_index=True); g()\n delete(train_bkt, vldt_bkt, test_bkt)\n print(log_template.format(bkt_id, df_train.shape, df_vldt.shape, df_test.shape))\n df_test.set_index('click_id', drop=True, inplace=True)\n df_test.sort_index(axis=0, inplace=True)\n return df_train, df_vldt, df_test\n\ndef feature_target_split(df, target_col_name, inplace=False):\n y = df[target_col_name]; g()\n if True == inplace:\n df.drop(target_col_name, axis=1, inplace=True); g()\n X = df\n else:\n X = df.drop(target_col_name, axis=1, inplace=True); g()\n return X, y\n\ndef prep_feature_target_full_data(): \n df_train, df_vldt, df_test = prep_data_set_full_data()\n X_train, y_train = feature_target_split(df_train, target_col_name = 'is_attributed', inplace = True)\n X_vldt, y_vldt = feature_target_split(df_vldt, target_col_name = 'is_attributed', inplace = True)\n delete(df_train, df_vldt)\n print('----: prep_feature_target_full_data :-----')\n print('-- X_train={}; y_train={}; X_vldt={}; y_vldt={}; df_test={}'.format(\n X_train.shape, y_train.shape, X_vldt.shape, y_vldt.shape, df_test.shape))\n print('-- features :', X_train.columns.values.tolist())\n print('-- categorical :', g_fit_params['categorical_feature'])\n print('-- y_train.value_counts\\n:', y_train.value_counts())\n print('-- y_vldt.value_countsl\\n:', y_vldt.value_counts())\n return X_train, y_train, X_vldt, y_vldt, df_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model and Parameters\n\n## Model Parameters \n\n

    parameters of LGBMClassifier constructor parameter. there are some dicts to manage baseline model and experimental parameters (grid-search) of each baseline

    \n\n> 模型参数，用来追踪基线模型（base-line model）以及在基线基础上做实验（experimental model）用到的网格搜索参数"},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\nfrom lightgbm import LGBMClassifier\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.model_selection import GridSearchCV\n\n# Light Gradient Boosting Classifier Parameters\n# https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html\n# https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html\n# https://lightgbm.readthedocs.io/en/latest/Parameters.html\n\ndef default_model(): \n lgb_default = LGBMClassifier()\n return lgb_default.set_params(\n objective = 'binary', # for binary classification\n metric = 'auc', # use metric required by this contest\n boosting_type = 'gbdt', # gbdt, dart, goss, rf\n verbose = 1, # -1\n nthread = 4, \n iid = False, # return average score across folds (not weighted)\n two_round = True \n )\n\ndef gbtd_base_001():\n return default_model().set_params(\n subsample = 0.8, # alias of bagging_fraction\n subsample_freq = 1, # alias of bagging_freq\n subsample_for_bin = 200000, # alias of bin_construct_sample_cnt\n colsample_bytree = 0.8, # alias of feature_fraction\n learning_rate = 0.08, \n num_leaves = 105, # 100:v103 -> 105:v105\n max_depth = 7,\n min_split_gain = 0.3, \n max_bin = 255, # default 255\n reg_alpha = 0.3, \n n_estimators = 2500 # alias of num_boost_round\n )\n\ng_base_models = {\n 'gbdt_base_001' : gbtd_base_001()\n}\n\ng_search_params = {\n # https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html\n # 'gbdt_base_001_exp_001' : {'num_leaves':[90, 95, 100, 105]}, # 105\n # 'gbdt_base_001_exp_003' : {'learning_rate':[0.1, 0.08, 0.06]}, # 0.8 is the best\n # 'gbdt_base_001_exp_004' : {'max_bin':[255, 315]}, # no difference\n 'gbdt_base_001_exp_002' : {'min_sum_hessian_in_leaf':[0.001, 0.01, 0.05, 0.1]}, \n 'gbdt_base_001_exp_005' : {'min_split_gain':[0.3, 0.4, 0.5]},\n 'gbdt_base_001_exp_010' : {'num_leaves':[90,105], 'min_split_gain':[0.3,0.4]}\n}\n\ndef update_data_balancing_param(lgb_model, y_value_counts, majority_val=0, minority_val=1):\n print('y_value_counts: \\n{}'.format(y_value_counts))\n if g_scale_pos_weight is None:\n print(f'use global config: scale_pos_weight={g_scale_pos_weight}')\n lgb_model.set_params(scale_pos_weight = g_scale_pos_weight)\n else :\n unbalance_degree = y_value_counts[majority_val] / y_value_counts[minority_val]\n print(f'majority_count/minority_count={unbalance_degree}') \n if unbalance_degree > 1.5 or unbalance_degree < 0.66:\n print(f'unbalance_dgree')\n lgb_model.set_params(is_unbalance = True)\n return lgb_model\n\ndef get_model_and_search_params(base_model_id, search_params_id):\n return g_base_models[base_model_id], g_search_params[search_params_id]\n\n# test\n# model, search_params = get_model_and_search_params('gbdt_base_001', 'gbdt_base_001_exp_001')\n# print(model.get_params(), '\\n', search_params)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fit-Parameters \n\nparameters of LGBMClassifier.fit() function"},{"metadata":{"trusted":true},"cell_type":"code","source":"g_fit_params = {\n 'categorical_feature' : g_categorical_features,\n 'early_stopping_rounds' : 25,\n 'verbose' : 10,\n 'eval_metric' : 'auc'\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Function for fitting model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit_model(X_train, y_train, X_validate, y_validate, lgb_model): \n '''\n id_in_g_base_models: hash key in g_base_models, such as 'gbdt_base_001'\n '''\n # fit parameters\n cat_fea_indices = list(map(lambda col:X_train.columns.get_loc(col), g_fit_params['categorical_feature']))\n print('------ fit parameters: ---------------------')\n print('early_stopping_rounds: {}'.format(g_fit_params['early_stopping_rounds']))\n print('verbose: {}'.format(g_fit_params['verbose']))\n print('eval_metric: {}'.format(g_fit_params['eval_metric']))\n print('categorical_feature: ', cat_fea_indices)\n for i in cat_fea_indices:\n print(\"\\t\", i, \":\", g_fit_params['categorical_feature'][i])\n print('------ updarte balancing parameters: ---------------------')\n lgb_model = update_data_balancing_param(lgb_model, y_train.value_counts())\n print('------ model parameters: ---------------------') \n print(lgb_model.get_params())\n # fit model\n fitted = lgb_model.fit(X_train, y_train \n , categorical_feature = cat_fea_indices \n , early_stopping_rounds = g_fit_params['early_stopping_rounds']\n , verbose = g_fit_params['verbose']\n , eval_metric = g_fit_params['eval_metric']\n , eval_set = [(X_validate, y_validate)])\n # return\n g()\n return fitted","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fit Model\n\nfetch a baseline model \n\n> 选取一个基线模型"},{"metadata":{"trusted":true},"cell_type":"code","source":"g_base_model = g_base_models['gbdt_base_001']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"prepare dataset， including train-set, validate-set, test-set(for submit)\n\n> 准备数据集，包括：训练集、验证集、以及用于提交模型结果的测试集"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"with timer_memory('prep_feature_target_full_data'):\n X_train, y_train, g_X_vldt, g_y_vldt, g_df_test = prep_feature_target_full_data()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"fit model\n> 训练模型"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict and Submit\n\npredict on test-set and generate submission file\n> 在测试集上做预测，并生成提交文件"},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_and_submit(model_fitted, num_iteration):\n sub = pd.DataFrame()\n sub['click_id'] = g_df_test.index\n sub['click_id'] = sub['click_id'].astype('int')\n pred_prob = model_fitted.predict_proba(X=g_df_test, num_iteration=num_iteration)\n sub['is_attributed'] = pred_prob[:,1].reshape(-1,1)\n sub.to_csv('submit.csv', index=False, float_format='%.9f')\n print(f'num_iteration: {num_iteration}')\n print(f'sub.shape: {sub.shape}')\n return sub","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with timer_memory('predict_and_sumit'):\n submit = predict_and_submit(g_model_fitted, g_model_fitted.best_iteration_)\n submit.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluate Model\n\nfunctions for evaluating model\n\n> 用于评估模型效果的函数\n\n```python\ndef plot_roc_auc_curve(y_label, y_proba_pred)\ndef plot_pr_curve(y_label, y_proba_pred)\ndef plot_confusion_matrix(y_label, y_label_pred)```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_roc_auc_curve(y_label, y_proba_pred): \n from sklearn.metrics import roc_auc_score\n from sklearn.metrics import roc_curve\n import matplotlib.pyplot as plt \n auc_score = roc_auc_score(y_label.values, y_proba_pred)\n fpr, tpr, thresh = roc_curve(y_label.values, y_proba_pred, pos_label=None)\n plt.figure(figsize=(6,6))\n plt.title('ROC curve')\n plt.plot(fpr, tpr, label='auc score: {:.4f}'.format(auc_score))\n plt.plot([0, 1], [0, 1], 'k--')\n plt.axis([-0.01, 1, 0, 1])\n plt.xlabel('False Positive Rate')\n plt.ylabel('True Positive Rate')\n plt.annotate('Minimum ROC Score of 50% \\n (This is the minimum score to get)'\n ,xy=(0.5, 0.5), xytext=(0.6, 0.3), arrowprops=dict(facecolor='#6E726D', shrink=0.05))\n plt.legend()\n plt.show()\n\ndef plot_pr_curve(y_label, y_proba_pred):\n import matplotlib.pyplot as plt\n from sklearn.metrics import precision_recall_curve\n precision, recall, threshold = precision_recall_curve(y_label.values, y_proba_pred)\n plt.figure(figsize=(6,6))\n plt.title('precision-recall curve', fontsize=16)\n plt.plot(recall, precision, 'b-', linewidth=2)\n plt.plot([0, 1], [0, 1], 'r--')\n plt.xlabel('precision')\n plt.ylabel('recall')\n plt.axis([-0.01,1,0,1])\n plt.show()\n\ndef plot_confusion_matrix(y_label, y_label_pred):\n import matplotlib.pyplot as plt\n from sklearn.metrics import confusion_matrix\n cfsn_matrix = confusion_matrix(y_label.values, y_label_pred)\n fig, ax = plt.subplots(1, 1,figsize=(6,6))\n sns.heatmap(cfsn_matrix, annot=True, cmap=plt.cm.copper)\n ax.set_title(\"confusion matrix\")\n ax.set_xticklabels(['0', '1'], rotation=90)\n ax.set_yticklabels(['0', '1'], rotation=360)\n ax.set_xlabel('predict labels')\n ax.set_ylabel('true labels')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"in order to reflect the prediction ability in production environment, label distribution must be the same as in origional data-set (no downsampling)\n\n> 为了真实反应生产环境的预测效果，用于评估的数据集必须与原始数据集标签分布一致（不能经过降采样）"},{"metadata":{"trusted":true},"cell_type":"code","source":"# data-set must be original un-balanced data, can not be under-sampled\ng_y_pred_proba_vldt = g_model_fitted.predict_proba(X=g_X_vldt, num_iteration=g_model_fitted.best_iteration_)[:,1]\ng_y_pred_label_vldt = g_model_fitted.predict(X=g_X_vldt, num_iteration=g_model_fitted.best_iteration_)\ng_y_vldt.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"evaluate models\n\n> 评估模型"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_roc_auc_curve(g_y_vldt, g_y_pred_proba_vldt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_pr_curve(g_y_vldt, g_y_pred_proba_vldt)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import classification_report\nreport = classification_report(g_y_vldt, g_y_pred_label_vldt, target_names=['is_not_attributed','is_attributed'])\nprint(report)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_confusion_matrix(g_y_vldt, g_y_pred_label_vldt)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot Features\n\nfunctions for plotting features\n\n> 用于评估特征的函数\n\n```python\ndef plot_num_fea_distribution(df, categorical=g_categorical_features)\ndef headmap_plot_fea_target_corr_matrix(df, categorical = g_categorical_features)\ndef boxplot_of_fea_target_corr(df, categorical = g_categorical_features)```"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def headmap_plot_fea_target_corr_matrix(df, categorical = g_categorical_features):\n corr = df.drop(g_categorical_features, axis=1).corr()\n ax = sns.heatmap(corr, cmap='coolwarm_r', annot_kws={'size':20})\n ax.set_title('Correlation Matrix \\n', fontsize=14)\n plt.show(); g()\n\ndef plot_num_fea_distribution(df, categorical = g_categorical_features):\n feature_names = df.columns.values.tolist()\n for cat_fea_name in categorical + ['is_attributed']:\n if cat_fea_name in feature_names:\n feature_names.remove(cat_fea_name)\n print(\"numerical features: {}\".format(feature_names))\n fig, axs = plt.subplots(ncols=2, nrows=0, figsize=(12, 36))\n plt.subplots_adjust(right=2)\n plt.subplots_adjust(top=2)\n sns.color_palette(\"husl\", 8)\n sns.set_style(\"white\")\n sns.set_color_codes(palette='deep')\n for i, fea_name in enumerate(list(df[feature_names]), 1):\n plt.subplot(len(list(feature_names)), 2, i)\n sns.distplot(df[fea_name], fit=norm, color=\"b\"); \n (mu, sigma) = norm.fit(df[fea_name]) \n plt.legend([\n 'Normal dist. ($\\mu=$ {:.2f} and $\\sigma=$ {:.2f} )'.format(mu, sigma)\n ],loc='best') \n plt.xlabel('{}'.format(fea_name), size=15, labelpad=12.5)\n plt.ylabel(\"Frequency\", size=15, labelpad=12.5)\n for j in range(2):\n plt.tick_params(axis='x', labelsize=12)\n plt.tick_params(axis='y', labelsize=12)\n sns.despine(trim=True, left=True)\n plt.show(); g()\n\ndef boxplot_of_fea_target_corr(df, categorical = g_categorical_features):\n feature_names = df.columns.values.tolist()\n if 'is_attributed' in feature_names:\n feature_names.remove('is_attributed')\n print(\"feature_count: {}\".format(len(feature_names))) \n fig, axs = plt.subplots(ncols=2, nrows=0, figsize=(12, 36))\n plt.subplots_adjust(right=2)\n plt.subplots_adjust(top=2)\n sns.color_palette(\"husl\", 8)\n for i, fea_name in enumerate(list(df[feature_names]), 1):\n if fea_name.startswith(\"cumcount_\"):\n outlier_plot_threshold = 18\n df.loc[df[fea_name]>outlier_plot_threshold, (fea_name)] = outlier_plot_threshold\n plt.subplot(len(list(feature_names)), 2, i)\n if fea_name in categorical:\n #sns.countplot(x=fea_name, hue='is_attributed', data=df)\n sns.violinplot(y=fea_name, x='is_attributed', data=df)\n else:\n sns.boxplot(y=fea_name, x='is_attributed', data=df)\n plt.xlabel('{}'.format('is_attributed'), size=15, labelpad=12.5)\n plt.ylabel(fea_name, size=15, labelpad=12.5)\n for j in range(2):\n plt.tick_params(axis='x', labelsize=12)\n plt.tick_params(axis='y', labelsize=12)\n plt.legend(loc='best', prop={'size': 10})\n plt.show(); g()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"plot feature importance\n\n> 评估特征在lightgbm模型训练过程中的重要程度"},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb.plot_importance(g_model_fitted.booster_, importance_type='split')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

    need a down-sampled data set to plot features, this has 2 benifits:

    \n1. virtualizing effect of both majority examples and minority examples: if no down-sampling, almost all input to plotting functions will be majority examples\n2. it is more close with the model training input, which is also down-sampled\n\n> 需要一个降采样之后的数据集来可视化特征分布：一方面不平衡数据集会导致可视化无法体现正例负例之间的差异（送给可视化函数的特征几乎全部都是负例(majority examples))；另一方面降采样是的可视化结果与模型训练一致（用于训练模型的特征也是经过降采样的特征）"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"df_plot = random_down_sample(\n g_X_vldt.merge(g_y_vldt, left_index=True, right_index=True), \n majority_multiply=g_majority_multiply, target_col_name='is_attributed', \n minority_val = 1, majority_val = 0); g()\ndf_plot['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"plot features, including feature distribution, "},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"plot_num_fea_distribution(df_plot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"headmap_plot_fea_target_corr_matrix(df_plot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"boxplot_of_fea_target_corr(df_plot)\ndelete(df_plot)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"with timer_memory('plot_metric_auc'):\n lgb.plot_metric(g_model_fitted, 'auc')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```python\ndef plot_learning_curve(model, X, y, cv, ylim=None, n_jobs=-1, train_sizes=np.array([0.1,0.5,0.75,1])):```"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_learning_curve(model, X, y, cv, ylim=None, n_jobs=-1, train_sizes=np.array([0.1,0.5,0.75,1])):\n import matplotlib.pyplot as plt\n from sklearn.model_selection import learning_curve\n fig, ax = plt.subplots(1, 1,figsize=(10,6))\n if ylim is not None:\n plt.ylim(*ylim)\n train_sizes, train_scores, test_scores = learning_curve(\n model, X, y, cv=cv, shuffle=False, n_jobs=n_jobs, train_sizes=train_sizes)\n train_scores_mean = np.mean(train_scores, axis=1)\n train_scores_std = np.std(train_scores, axis=1)\n test_scores_mean = np.mean(test_scores, axis=1)\n test_scores_std = np.std(test_scores, axis=1)\n ax.fill_between(train_sizes, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.1, color=\"#ff9124\")\n ax.fill_between(train_sizes, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.1, color=\"#2492ff\")\n ax.plot(train_sizes, train_scores_mean, 'o-', color=\"#ff9124\", label=\"Training score\")\n ax.plot(train_sizes, test_scores_mean, 'o-', color=\"#2492ff\", label=\"Cross-validation score\")\n ax.set_title(\"Learning Curve\", fontsize=14)\n ax.set_xlabel('Training size (m)')\n ax.set_ylabel('Score')\n ax.grid(True)\n ax.legend(loc=\"best\")\n plt.show()\n\nfrom sklearn.model_selection import PredefinedSplit\ndef get_cv_splitter(y_train:pd.Series):\n total_size, test_size = y_train.size, y_train.size // 10\n test_fold = np.full(shape=(total_size,), fill_value=-1, dtype='int')\n test_fold[-1*test_size:] = 0\n cv = PredefinedSplit(test_fold=test_fold)\n print(f'total_size={total_size}; test_size={test_size}')\n return cv","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"#with timer_memory('plot_learning_curve'):\n# plot_learning_curve(g_base_model, X=X_train, y=y_train, cv=get_cv_splitter(y_train), ylim=(0.92, 1.01), scoring='roc_auc')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Grid Search\n\n

    existing function used:

    \n\n```python\ng_base_models = {\n 'gbdt_base_001' : gbtd_base_001(), ...\n}\ng_search_params = {\n # 'gbdt_base_001_exp_001' : {'num_leaves':[90, 95, 100, 105]}, # 105\n # 'gbdt_base_001_exp_003' : {'learning_rate':[0.1, 0.08, 0.06]}, # 0.8 is the best\n # 'gbdt_base_001_exp_004' : {'max_bin':[255, 315]}, # no difference\n 'gbdt_base_001_exp_002' : {'min_sum_hessian_in_leaf':[0.001, 0.01, 0.05, 0.1]}, \n 'gbdt_base_001_exp_005' : {'min_split_gain':[0.3, 0.4, 0.5]},\n 'gbdt_base_001_exp_010' : {'num_leaves':[90,105], 'min_split_gain':[0.3,0.4]}\n}\ndef get_model_and_search_params(base_model_id, search_params_id)\ndef update_data_balancing_param(lgb_model, y_value_counts, majority_val=0, minority_val=1)\n```\n\n

    new function defined as belows:

    \n```python\ndef grid_search(X_train, y_train, X_vldt, y_vldt, base_model, param_grid)```\n

    parameters turning:

    \n* [https://lightgbm.readthedocs.io/en/latest/Features.html#leaf-wise-best-first-tree-growth](https://lightgbm.readthedocs.io/en/latest/Features.html#leaf-wise-best-first-tree-growth)\n* [https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\nfrom sklearn.model_selection import ParameterGrid\nfrom sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score, accuracy_score, classification_report\n\ndef grid_search(X_train, y_train, X_vldt, y_vldt, base_model, param_grid):\n # result\n estimator_lst,grid_point_lst,precision_lst,recall_lst,f1_lst,auc_lst=[],[],[],[],[],[]\n # grid search\n for grid_point in list(ParameterGrid(param_grid)):\n # model parameters\n print(f'----- grid_point: \\n{grid_point}')\n base_model = base_model.set_params(**grid_point)\n base_model = base_model.set_params(silent = True)\n base_model = base_model.set_params(verbosity = 0)\n print('update_data_balancing_param:' )\n base_model = update_data_balancing_param(base_model, y_train.value_counts())\n print(f'model parameters: \\n{base_model.get_params()}')\n # fit parameters\n print('cat_fea: {}'.format(g_fit_params['categorical_feature']))\n print('fea: {}'.format(X_train.columns))\n print(type(X_train.columns))\n print(X_train.columns.get_loc('app'))\n cat_fea_indices = list(map(lambda col:X_train.columns.get_loc(col), g_fit_params['categorical_feature']))\n fit_params_copy = g_fit_params.copy()\n fit_params_copy['categorical_feature'] = cat_fea_indices\n fit_params_copy['verbose'] = -1\n print(f'fit_params_copy: \\n{fit_params_copy}')\n # fit\n fitted = base_model.fit(\n X_train, y_train, eval_set = [(X_vldt, y_vldt)], **fit_params_copy); g()\n # pred\n y_pred = fitted.predict(X_vldt)\n # append_score\n grid_point_lst.append(grid_point) \n estimator_lst.append(fitted)\n precision_lst.append(precision_score(y_vldt, y_pred))\n recall_lst.append(recall_score(y_vldt, y_pred))\n f1_lst.append(f1_score(y_vldt, y_pred))\n auc_lst.append(roc_auc_score(y_vldt, y_pred))\n # search result\n score_dict = pd.DataFrame(data={\n 'precision':precision_lst, # TP/(TP+FP)\n 'recall':recall_lst, # TP/(TP+FN)\n 'f1':f1_lst, # F1 Score\n 'auc':auc_lst # AUC\n }, index=grid_point_lst)\n estimator_dict = pd.DataFrame(data={'estimator':estimator_lst}, index=grid_point_lst)\n return score_dict, estimator_dict\n\ndef run_grid_search(base_model_id, exp_params_id): \n with timer_memory('grid_search %'.format(exp_params_id)):\n lgb_model, search_params = get_model_and_search_params(base_model_id, exp_params_id)\n score_dict, estimator_dict = grid_search(X_train, y_train, g_X_vldt, g_y_vldt, lgb_model, search_params)\n return score_dict","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run Grid Search"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"g_grid_search_results = {\n 'gbdt_base_001_exp_002' : run_grid_search('gbdt_base_001', 'gbdt_base_001_exp_002'),\n 'gbdt_base_001_exp_005' : run_grid_search('gbdt_base_001', 'gbdt_base_001_exp_005'),\n 'gbdt_base_001_exp_010' : run_grid_search('gbdt_base_001', 'gbdt_base_001_exp_010')\n}\n\ndef plot_grid_search_score(exp_id, grid_results=g_grid_search_results):\n grid_results[exp_id].plot.barh()\n return grid_results[exp_id]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Grid Search Results"},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"plot_grid_search_score('gbdt_base_001_exp_002')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"plot_grid_search_score('gbdt_base_001_exp_005')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"plot_grid_search_score('gbdt_base_001_exp_010')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}