{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":8540,"databundleVersionId":862041,"sourceType":"competition"},{"sourceId":172527488,"sourceType":"kernelVersion"},{"sourceId":172527505,"sourceType":"kernelVersion"}],"dockerImageVersionId":29662,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\n

This is an task to predict whether the APP will be installed (is_attributed=True) after user click the AD. The data-set is un-balanced (`positive/negative = 0.03/1`) and is too big for single-server environment. Belows are the challenge and solutions during the model training.

\n\n* **Memore Insufficient when Data Loading**: the data-set is too large to load into memory, thus the data is partioned before feature-extraction and down-sampling. And then the down-sampled data are merged together as the training set
\n* **Model Evaluation**: If fill down-sampled data into the verification-set, it can not reflect the real performace in production. Thus the original data, i.e non-down-sampled data, is used as the verification set in parameter search and model evaluation. Also a function is writen to support this kind of parameter search\n* **Parameter Change Trace**: Simple common code is implemented to trace base-line models and experiments (parameter-serch) base on these baselines.\n* **Feature Selections**: basically is based on the feature-importency. Since the testing-set is hour-based (only several hour data), the day-based feature will be in-consistent between training set and testing set. Thus, the features are only aggregated by hour or by all-data, not by day. \n","metadata":{}},{"cell_type":"markdown","source":"# libraries and Util Functions","metadata":{}},{"cell_type":"code","source":"# python util\nimport os\nimport gc\nfrom operator import methodcaller\n\n# data processing\nimport numpy as np\nimport pandas as pd\nfrom sklearn.preprocessing import RobustScaler as RobustScaler\nfrom scipy.stats import skew, norm\nfrom scipy.stats import boxcox_normmax, boxcox\nfrom scipy.special import boxcox1p\n\n# lightgbm\nimport lightgbm as lgb\nfrom lightgbm import LGBMClassifier\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.model_selection import GridSearchCV\n\n# plot\nimport matplotlib.pyplot as plt \nimport seaborn as sns \n\n# random seed\n# import random\n# random.seed(567)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:41:15.106005Z","iopub.execute_input":"2024-04-19T01:41:15.106272Z","iopub.status.idle":"2024-04-19T01:41:16.487691Z","shell.execute_reply.started":"2024-04-19T01:41:15.106231Z","shell.execute_reply":"2024-04-19T01:41:16.486982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Util funtions \n\n```python\ng_enable_log = True\ndef log(log_str)\ndef g() #short-cut of gc.collect()\ndef delete(*obj_list)\ndef robust_boxcox(data:pd.Series, lmbda=None)\ndef robust_inv_boxcox(data:pd.Series, lmbda)\n@contextmanager\ndef timer_memory(name)\n```","metadata":{}},{"cell_type":"code","source":"g_enable_log = True\ndef log(log_str):\n if g_enable_log:\n print(log_str)\n\ndef g():\n return gc.collect()\n\ndef delete(*obj_list):\n for obj in obj_list:\n del obj\n gc_cnt = g()\n if gc_cnt > 0:\n log(\"unreachable_obj_found: {}\".format(gc_cnt))\n\n#reference:\n#https://stackoverflow.com/questions/47213443/divide-by-zero-encountered-in-log-scipy-stats-boxcox\ndef init_robust_boxcox():\n from scipy.special import inv_boxcox\n from scipy.stats import boxcox\n\n def robust_boxcox(data:pd.Series, lmbda=None):\n if lmbda is None:\n transformed, lmbda = boxcox(1 + data)\n # enhance in condition that boxcox returns \n # a extremely close-to-zero negative float instead of return 0\n if lmbda <= 0:\n lmbda = 0\n transformed = np.log1p(data) #log(1+data)\n return (transformed, lmbda)\n else:\n if lmbda <= 0:\n lmbda = 0\n transformed = np.log1p(data) #log(1+data)\n return (transformed, lmbda)\n else:\n transformed = boxcox(1 + data, lmbda)\n return (transformed, lmbda)\n\n def robust_inv_boxcox(data:pd.Series, lmbda):\n if lmbda <= 0:\n return np.expm1(data) #inv of log1p\n else:\n transformed = inv_boxcox(data, lmbda) - 1\n return transformed\n \n return robust_boxcox, robust_inv_boxcox\n\nrobust_boxcox, robust_inv_boxcox = init_robust_boxcox()\n\nimport time\nimport psutil\nfrom contextlib import contextmanager \n@contextmanager\ndef timer_memory(name):\n t0 = time.time()\n yield\n print(f'Memory: {(psutil.Process(os.getpid()).memory_info().rss/2**30):.02f}GB')\n print(f'{name} done in {time.time()-t0:.0f}s')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:41:16.489480Z","iopub.execute_input":"2024-04-19T01:41:16.489750Z","iopub.status.idle":"2024-04-19T01:41:16.504399Z","shell.execute_reply.started":"2024-04-19T01:41:16.489694Z","shell.execute_reply":"2024-04-19T01:41:16.503703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Input Files\n\n

Input Files

\n
  • train.csv: training set, the 1st column is IP, records are sorted by click_time
    • \n
  • test.csv: testing set, the 1st column is click_id (from 0 to record_num-1), 2nd column is IP, sorted by click_time
    • \n

    ","metadata":{}},{"cell_type":"code","source":"# for dirname, _, filenames in os.walk('/kaggle/input'):\n# for filename in filenames:\n# print(os.path.join(dirname, filename))\n# !head -n3 /kaggle/input/talkingdata-adtracking-fraud-detection/train.csv\n# !head -n3 /kaggle/input/talkingdata-adtracking-fraud-detection/test.csv ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

    Data Partitioning

    \n1. split the dataset into 20 partitions according to the value of IP % 20, which is named as IPBucket\n2. each bucket is smaller enough to be loaded into memory, now we can add whatever features we want to try without worrying about the memory usage\n3. down-sample the training set to balance the data-set. Since majory(is_attributed=1)/minority(is_attributed=0) is 99.7/0.3, down-sample will reduce training set to 0.3% as before as the same time\n4. fit model with down-sampled training-set (the verification-set and testing-set won't be down-sampled)\n","metadata":{}},{"cell_type":"code","source":"# %%time\n# # create directories and make sure they are empty\n# !cd /kaggle/working/\n# !mkdir -p /kaggle/working/train /kaggle/working/test \n# !rm -f /kaggle/working/train/* /kaggle/working/test/*\n\n# # full data with 20 bucket \n# # write column names into each file\n# !for i in $(seq 0 19); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' > /kaggle/working/train/train_${i}.csv; done\n# !for i in $(seq 0 19); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' > /kaggle/working/test/test_${i}.csv; done\n\n# # split the data set according to IP(1st column of train.csv, 2nd column of test.csv) % 20\n\n# # Splits the data based on the first column ($1) modulo 20 and appends each line to the corresponding file in the /kaggle/working/train/ directory.\n# !cat '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' | grep -v '^[a-zA-Z]' | awk -F\",\" '{print $0 >> \"/kaggle/working/train/train_\"$1%20\".csv\"}'\n# # Splits the data based on the first column ($2) modulo 20 and appends each line to the corresponding file in the /kaggle/working/train/ directory.\n# !cat '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' | grep -v '^[a-zA-Z]' | awk -F\",\" '{print $0 >> \"/kaggle/working/test/test_\"$2%20\".csv\"}'\n\n# # configurations for next steps\n# g_ip_bkt_num = 20 # partition number\n# g_vldt_set_size = 5000000 # validation set size\n# g_is_down_sample = True # configration to enable the train-set down-sample\n# g_majority_multiply = 1 # configration to decide majority record numbers after down sample \n# # 1: means majority_record_number = minority_record_number\n# # 2: means majority_record_number = minority_record_number * 2\n# g_scale_pos_weight = 1 # positive example weight (use to set scale_pos_weight parameter of lightgbm)\n# # integer: positive example weight = g_s negative example\n# # none: won't set scale_pos_weight, \n# # but set is_unbalanced=True to let lightgbm decide the weight by itself\n# # check data\n# !head -n3 /kaggle/working/train/* /kaggle/working/test/* # first 3 lines of each file\n# !ls -lh /kaggle/working/train/* /kaggle/working/test/* # file size\n# !wc -l /kaggle/working/train/* /kaggle/working/test/* # record numbers of each file","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# configurations for next steps\ng_ip_bkt_num = 20 # partition number\ng_vldt_set_size = 5000000 # validation set size\ng_is_down_sample = True # configration to enable the train-set down-sample\ng_majority_multiply = 1 # configration to decide majority record numbers after down sample \n # 1: means majority_record_number = minority_record_number\n # 2: means majority_record_number = minority_record_number * 2\ng_scale_pos_weight = 1 # positive example weight (use to set scale_pos_weight parameter of lightgbm)\n # integer: positive example weight = g_s negative example\n # none: won't set scale_pos_weight, \n # but set is_unbalanced=True to let lightgbm decide the weight by itself","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:26:45.462437Z","iopub.execute_input":"2024-04-17T09:26:45.462769Z","iopub.status.idle":"2024-04-17T09:26:45.467690Z","shell.execute_reply.started":"2024-04-17T09:26:45.462725Z","shell.execute_reply":"2024-04-17T09:26:45.466823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If don't want to partition these data, or just want to use a small dataset to debug the program, we can un-comment the code below.","metadata":{}},{"cell_type":"code","source":"#%%time\n#!cd /kaggle/input/\n#!mkdir -p /kaggle/input/train /kaggle/input/test \n#!rm -f /kaggle/input/train/* /kaggle/input/test/*\n\n#debug with 45000 data with 1 bucket\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' > /kaggle/input/train/train_${i}.csv; done\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' > /kaggle/input/test/test_${i}.csv; done\n#!tail -n 45000 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' | grep -v '^[a-zA-Z]' | awk -F\",\" '{print $0 >> \"/kaggle/input/train/train_\"$1%1\".csv\"}'\n#!cat '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' | grep -v '^[a-zA-Z]' |awk -F\",\" '{print $0 >> \"/kaggle/input/test/test_\"$2%1\".csv\"}'\n#g_ip_bkt_num = 1\n#g_vldt_set_size = 50 \n#g_is_down_sample = True\n#g_scale_pos_weight = 1\n#g_majority_multiply = 1\n\n#latest 30000000 data with 1 bucket\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' > /kaggle/input/train/train_${i}.csv; done\n#!for i in $(seq 0 0); do head -n1 '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' > /kaggle/input/test/test_${i}.csv; done\n#!tail -n 30000000 '/kaggle/input/talkingdata-adtracking-fraud-detection/train.csv' | grep -v '^[a-zA-Z]' >> \"/kaggle/input/train/train_0.csv\" \n#!cat '/kaggle/input/talkingdata-adtracking-fraud-detection/test.csv' | grep -v '^[a-zA-Z]' >> \"/kaggle/input/test/test_0.csv\"\n#g_ip_bkt_num = 1\n#g_vldt_set_size = 5000000\n#g_is_down_sample = False\n#g_scale_pos_weight = 99.7\n#g_majority_multiply = 1\n\n#check data\n#!head -n3 /kaggle/input/train/* /kaggle/input/test/*\n#!ls -lh /kaggle/input/train/* /kaggle/input/test/*\n#!wc -l /kaggle/input/train/* /kaggle/input/test/*","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering\n\n```python\n# feature_name -> scaler\ng_scaler_dict = {}\n# feature_name -> lmbda\ng_boxcox_lmbda_dict = {} \ndef box_cox_trans(df, fea_name, sv_policy)\ndef scaler_trans(df, fea_name, sv_policy, scaler)```","metadata":{}},{"cell_type":"code","source":"g_scaler_dict = {} # feature_name -> scaler\ng_boxcox_lmbda_dict = {} # feature_name -> lmbda\n\ndef box_cox_trans(df, fea_name, sv_policy):\n '''sv_policy (lmbda saving policy): new_and_save, reuse, new'''\n df[fea_name] = df[fea_name].astype('float64'); g()\n df.loc[df[fea_name] <= 0, (fea_name)] = 0.000001\n if sv_policy == 'new_and_save':\n df[fea_name], lmbda = robust_boxcox(df[fea_name]); g()\n g_boxcox_lmbda_dict[fea_name] = lmbda\n elif sv_policy == 'reuse':\n lmbda = g_boxcox_lmbda_dict[fea_name]\n df[fea_name], lmbda = robust_boxcox(df[fea_name], lmbda); g()\n else:\n df[fea_name], lmbda = robust_boxcox(df[fea_name]); g()\n log(\"\\tboxcox lmbda: {}\".format(lmbda))\n return df[fea_name].astype('float32')\n\ndef scaler_trans(df, fea_name, sv_policy, scaler):\n '''sv_policy (scaler saving policy): new_and_save, reuse, new''' \n df[fea_name] = df[fea_name].astype('float64'); g()\n if sv_policy == 'new_and_save':\n df[[fea_name]] = scaler.fit_transform(df[[fea_name]]); g()\n g_scaler_dict[fea_name] = scaler\n elif sv_policy == 'reuse':\n df[[fea_name]] = g_scaler_dict[fea_name].transform(df[[fea_name]]); g()\n else: #'new'\n df[[fea_name]] = scaler.fit_transform(df[[fea_name]]); g()\n return df[fea_name].astype('float32') ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:42:25.689670Z","iopub.execute_input":"2024-04-19T01:42:25.689992Z","iopub.status.idle":"2024-04-19T01:42:25.703763Z","shell.execute_reply.started":"2024-04-19T01:42:25.689944Z","shell.execute_reply":"2024-04-19T01:42:25.702873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

    functions for creating new features

    \n\n```python\ndef add_grp_count(df, group_columns, dtype='uint32', scl=None, bc=False, sv='reuse')\ndef add_grp_stat(df, group_columns, stat_column, stat_fun_name, dtype, scl=None, bc=False, sv='reuse')\ndef add_grp_nxt_clk_intv(df, group_columns, scl=None, bc=False, sv='reuse')```","metadata":{}},{"cell_type":"code","source":"# parameters of below functions:\n# scl : scaler object\n# bc : is_boxcox, whether enable boxcox transform\n# sv : how to get transformer when boxcox tranforming and scaling\n# 'new': create new transformer\n# 'new_and_save': create new transformer and save it for future\n# 'reuse': reuse existing transformer\n\ndef add_grp_count(df, group_columns, dtype='uint32', scl=None, bc=False, sv='reuse'):\n '''\n steps: \n 1. group DataFrame(df) by group_columns\n 2. get the group size\n 3. attach group size to each rows\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv \n # feature name\n feature_name = \"cnt_grp_by_\" + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n # group-count\n group_col_and_cnt = df.groupby(group_columns).size().astype(dtype); g()\n # format-conversion: rename() => đổi tên cột giá trị mới được aggregation (count) thành tên feature mới\n group_col_and_cnt = group_col_and_cnt.rename(feature_name).to_frame().reset_index(); g()\n # merge\n df_tmp = df.merge(group_col_and_cnt, on=group_columns, how='left'); delete(df)\n df = df_tmp\n # boxcox\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g()\n # scaler\n if scaler is not None: #1\n df[feature_name] = scaler_trans(df, feature_name, sv_policy=sv, scaler=scaler); g() \n # return\n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}')\n return df\n\ndef add_grp_stat(df, group_columns, stat_column, stat_fun_name, dtype, scl=None, bc=False, sv='reuse'):\n '''\n steps:\n 1. group DataFrame(df) by group_columns \n 2. apply stat_fun_name on stat_column in each group\n 3. attach stat_func_name's ouput to each rows\n stat_func_nane: \"cumcount\", \"var\", \"count\", \"nunique\"\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv \n # feature name\n feature_name = stat_fun_name + \"_on_\" + stat_column + \"_grp_by_\" + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n # group then stat in each group\n grouped = df[group_columns + [stat_column]].groupby(group_columns); g()\n stat_applied = methodcaller(stat_fun_name)(grouped[stat_column]).astype(dtype); g()\n delete(grouped)\n if stat_fun_name == \"var\":\n stat_applied = stat_applied.transform(lambda x: x.fillna(0)); g()\n # data frame conversion\n group_columns_and_stat_column = stat_applied.rename(feature_name).to_frame()\n delete(stat_applied)\n # merge\n if stat_fun_name == \"cumcount\": \n df[feature_name] = group_columns_and_stat_column[feature_name].astype(dtype); g()\n elif stat_fun_name in ['var', 'count', 'nunique']:\n group_columns_and_stat_column = group_columns_and_stat_column.astype(dtype); g()\n group_columns_and_stat_column = group_columns_and_stat_column.reset_index(); g()\n df_tmp = df.merge(group_columns_and_stat_column, on=group_columns, how='left')\n delete(df)\n df = df_tmp\n else:\n raise Exception('un-supported stat_fun_name: {}'.format(stat_fun_name)) \n # boxcox transform and scaler\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g()\n if scaler is not None: #2\n df[feature_name] = scaler_trans(df, feature_name, sv_policy=sv, scaler=scaler); g()\n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}') \n return df\n\ndef add_grp_nxt_clk_intv(df, group_columns, scl=None, bc=False, sv='reuse'):\n '''\n add column by grouping then calculate click interval \n between the record and it's next click in the same group\n steps:\n 1.group DataFrame(df) by group_columns \n 2.calculate next click_time in the same group for each given record\n 3.calculate intervial between current click and it's next click in the same group\n 4.merge the intervial time into the data set\n 5.scale the intervial time with RobustScaler if is_scale is True\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv\n # feature name\n feature_name = 'nxt_itvl_by_' + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n \n # Convert 'click_time' to datetime\n df['click_time'] = pd.to_datetime(df['click_time'])\n \n # interval to next click\n df['click_time_in_sec'] = (df['click_time'].astype(np.int64)//10**9).astype(np.int32) # This line converts the 'click_time' column in the DataFrame df to seconds. It first converts the 'click_time' values to nanoseconds (1 second = 10^9 nanoseconds), then converts them to 32-bit integers and assigns the result to a new column named 'click_time_in_sec'.\n df[feature_name] = (df.groupby(group_columns)['click_time_in_sec'].shift(-1) - df['click_time_in_sec']).astype(np.float32); g() # taking the difference between the 'click_time_in_sec' of the current row and the next row within each group specified by 'group_columns'. The shift(-1) function shifts the 'click_time_in_sec' values one position up within each group\n df[feature_name] = df[feature_name].fillna(df[feature_name].mean()); g() \n print('\\tfillna: {}'.format(df[feature_name].mean()))\n df.drop(['click_time_in_sec'], axis=1, inplace=True)\n # boxcox transform and scaler\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g()\n if scaler is not None: #3\n df[feature_name] = scaler_trans(\n df, feature_name, sv_policy=sv, scaler=scaler); g()\n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}') \n return df\n\ndef add_grp_pre_clk_intv(df, group_columns, scl=None, bc=False, sv='reuse'):\n '''\n add column by grouping then calculate click interval \n between the record and it's previouse click in the same group\n steps:\n 1.group DataFrame(df) by group_columns \n 2.calculate previous click_time in the same group for each given record\n 3.calculate the interval between click_time of the record \n and it's previouse click_time in the same group\n 4.merge the intervial time into the data set\n 5.scale the intervial time with RobustScaler if is_scale is True\n '''\n # parameters\n scaler = scl\n is_boxcox = bc\n save_transformer = sv \n # feature name\n feature_name = 'pre_itvl_by_' + \"_\".join(group_columns)\n print(\"add feature: \", feature_name)\n \n # Convert 'click_time' to datetime\n df['click_time'] = pd.to_datetime(df['click_time'])\n \n # calculate click interval\n df['click_time_in_sec'] = (df['click_time'].astype(np.int64)//10**9).astype(np.int32)\n df[feature_name] = (df['click_time_in_sec'] - df.groupby(group_columns)['click_time_in_sec'].shift(1)).astype(np.float32); g()\n df[feature_name] = df[feature_name].fillna(df[feature_name].mean()); g()\n print('\\tfillna: {}'.format(df[feature_name].mean()))\n df.drop(['click_time_in_sec'], axis=1, inplace=True)\n # boxcox transform and scaler\n if is_boxcox:\n df[feature_name] = box_cox_trans(df, feature_name, sv_policy=sv); g() \n if scaler is not None: #4\n df[feature_name] = scaler_trans(df, feature_name, sv_policy=sv, scaler=scaler); g() \n print(f'\\t{feature_name}: max={df[feature_name].max()}; min={df[feature_name].min()}; mean={df[feature_name].mean()}') \n return df\n\n# reference\n# help(pandas._libs.tslibs.timedeltas.Timedelta)\n# help(pandas._libs.tslibs.nattype.NaTType)\n# https://kapeli.com/dash_share?docset_file=Pandas&docset_name=Pandas&path=doc/reference/series.html%23timedelta-properties&platform=pandas&repo=Main&version=0.25.1","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:42:28.653210Z","iopub.execute_input":"2024-04-19T01:42:28.653518Z","iopub.status.idle":"2024-04-19T01:42:28.693164Z","shell.execute_reply.started":"2024-04-19T01:42:28.653463Z","shell.execute_reply":"2024-04-19T01:42:28.692350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def lda_topic(df, features):\n # Step 1: Combine the features into short text\n df['short_text'] = df[features].astype(str).agg(' '.join, axis=1)\n\n # Step 2: Feature Extraction\n count_vectorizer = CountVectorizer()\n short_text_matrix = count_vectorizer.fit_transform(df['short_text'])\n\n # Step 3: Topic Modeling with LDA\n num_topics = 5\n lda_model = LDA(n_components=num_topics)\n topics_of_records = lda_model.fit_transform(short_text_matrix)\n\n # Convert topics_of_records to DataFrame\n topic_columns = [f\"Topic_{i}\" for i in range(num_topics)]\n topics_df = pd.DataFrame(topics_of_records, columns=topic_columns)\n\n # Merge topics_of_records back to the original DataFrame\n df_with_topics = pd.concat([df, topics_df], axis=1)\n\n return df_with_topics","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:29.165401Z","iopub.execute_input":"2024-04-19T01:42:29.165714Z","iopub.status.idle":"2024-04-19T01:42:29.173578Z","shell.execute_reply.started":"2024-04-19T01:42:29.165658Z","shell.execute_reply":"2024-04-19T01:42:29.172701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.decomposition import LatentDirichletAllocation as LDA\nimport pandas as pd\n\ndef lda_topic_modelling(df, features):\n # Step 1: Combine the features into short text\n df['short_text'] = df[features].astype(str).agg(' '.join, axis=1)\n\n # Step 2: Feature Extraction\n count_vectorizer = CountVectorizer()\n short_text_matrix = count_vectorizer.fit_transform(df['short_text'])\n\n # Step 3: Topic Modeling with LDA\n num_topics = 5\n lda_model = LDA(n_components=num_topics)\n topics_of_records = lda_model.fit_transform(short_text_matrix)\n\n # Convert topics_of_records to DataFrame\n topic_columns = [f\"Topic_{i}\" for i in range(num_topics)]\n topics_df = pd.DataFrame(topics_of_records, columns=topic_columns)\n\n # Merge topics_of_records back to the original DataFrame\n df_with_topics = pd.concat([df, topics_df], axis=1)\n\n return df_with_topics\n\n# Example usage:\n# df = lda_topic_modelling(df, ['app', 'os', 'channel', 'device', 'ip'])\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:29.598073Z","iopub.execute_input":"2024-04-19T01:42:29.598365Z","iopub.status.idle":"2024-04-19T01:42:29.674033Z","shell.execute_reply.started":"2024-04-19T01:42:29.598323Z","shell.execute_reply":"2024-04-19T01:42:29.673181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def add_features(df, is_boxcox=False, is_scaler=False, save_transformer='reuse'):\n '''\n create extended features\n parameters:\n * save_transformer: how to get transformer when doing boxcox and scaling features\n * 'new': create new transformer\n * 'new_and_save': create new transformer and save it for future\n * 'reuse': reuse existing transformer\n '''\n # parameter short cut\n sv = save_transformer \n bc = is_boxcox\n s = None;\n if is_scaler:\n s = RobustScaler()\n # features and importancy\n # 2008: nxt_intv_by_ip_os_device_app\n df = add_grp_nxt_clk_intv(df, ['ip','os','device','app'], bc=None,scl=s,sv=sv);g()\n # 1747: channel\n # 1313: os\n # 1182: hh\n # 1051: app\n # 1035: cnt_grp_by_dd_hh_app_channel\n df = add_grp_count(df, ['dd','hh','app','channel'], bc=bc,scl=s,sv=sv);g()\n # 995 : cumcount_on_app_grp_by_ip_device_os\n df = add_grp_stat(df, ['ip','device','os'], 'app', 'cumcount', 'uint32', bc=bc,scl=s,sv=sv);g()\n # 990 : cnt_grp_by_ip_device\n df = add_grp_count(df, ['ip','device'], bc=bc,scl=s,sv=sv);g()\n # 929 : nunique_on_channel_by_ip\n df = add_grp_stat(df, ['ip'], 'channel', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g()\n # 921 : nunique_on_app_by_ip\n df = add_grp_stat(df, ['ip'], 'app', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g()\n # 835 : nxt_intv_by_ip_channel\n df = add_grp_nxt_clk_intv(df, ['ip','channel'], bc=None,scl=s,sv=sv);g()\n # 834 : cnt_grp_by_ip_hh_device\n df = add_grp_count(df, ['ip','hh','device'], bc=bc,scl=s,sv=sv);g()\n # 832 : nxt_intv_by_ip_app_channel\n df = add_grp_nxt_clk_intv(df, ['ip','app','channel'], bc=None,scl=s,sv=sv);g()\n # 825 : cnt_grp_by_app_channel\n df = add_grp_count(df, ['app','channel'], bc=bc,scl=s,sv=sv);g()\n # 716 : cnt_grp_by_ip_app\n df = add_grp_count(df, ['ip','app'], bc=bc,scl=s,sv=sv);g()\n # 635 : unique_on_app_grp_by_ip_hh\n df = add_grp_stat(df, ['ip','hh'], 'app', 'nunique','uint16',bc=bc,scl=s,sv=sv);g()\n # 592 : cnt_grp_by_ip_hh_app\n df = add_grp_count(df, ['ip','hh','app'], bc=bc,scl=s,sv=sv);g()\n # 432 : nunique_on_channel_by_app\n df = add_grp_stat(df, ['app'], 'channel', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g() \n # 422 : nunique_on_channel_by_hh_app\n df = add_grp_stat(df, ['hh','app'], 'channel', 'nunique', 'uint16',bc=None,scl=None,sv=sv);g() \n # 307 : device\n df = lda_topic_modelling(df, ['app', 'os', 'channel', 'device', 'ip'])\n return df","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:30.039595Z","iopub.execute_input":"2024-04-19T01:42:30.039906Z","iopub.status.idle":"2024-04-19T01:42:30.060995Z","shell.execute_reply.started":"2024-04-19T01:42:30.039856Z","shell.execute_reply":"2024-04-19T01:42:30.060173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

    function for random down-sampling, normal we setting majority_multiply=1 to make majority_examples_count/minority_examples_count=1, sometime we want more majority examples we can increase majority_multiply and vice versa

    \n\n```python\ndef random_down_sample(df, majority_multiply=1, target_col_name='is_attributed', minority_val = 1, majority_val = 0)```","metadata":{}},{"cell_type":"code","source":"def random_down_sample(df, majority_multiply=1, target_col_name='is_attributed', minority_val=1, majority_val=0):\n '''\n down sample the majority part, \n so that both part (target=1, target=0) has equal number of recordes\n '''\n minority_cnt = len((df[df[target_col_name] == minority_val])) \n majority_cnt = (minority_cnt * majority_multiply) // 1\n\n shuffled = df.sample(frac=1)\n minority = shuffled.loc[shuffled[target_col_name] == minority_val]\n majority = shuffled.loc[shuffled[target_col_name] == majority_val][:majority_cnt]\n\n undersample_df = pd.concat([minority, majority]).sample(frac=1)\n delete(shuffled, minority, majority)\n return undersample_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:42:30.909721Z","iopub.execute_input":"2024-04-19T01:42:30.910039Z","iopub.status.idle":"2024-04-19T01:42:30.917474Z","shell.execute_reply.started":"2024-04-19T01:42:30.909993Z","shell.execute_reply":"2024-04-19T01:42:30.916539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare Data Set\n\n## configures","metadata":{}},{"cell_type":"code","source":"# categorical features\n# g_categorical_features = ['app', 'device', 'os', 'channel', 'hh'] \ng_categorical_features = ['app', 'os', 'ip', 'device', 'channel'] \n# g_categorical_features = ['app', 'device', 'os', 'channel'] \n\n# only for extracting features, won't used in model training\n# g_non_train_columns = ['click_time', 'dd', 'ip']\ng_non_train_columns = ['click_time', 'dd']\n# g_non_train_columns = ['click_time', 'ip', 'attributed_time']\n\n# file spec\ndef get_file_spec(is_test_file):\n dtypes = {'ip':'uint32','app':'uint16','device':'uint8','os':'uint16',\n 'channel':'uint16','is_attributed':'int8','click_id':'int32'} \n date_columns = ['click_time']\n test_file_columns = ['click_time','ip','app','device','os','channel','click_id']\n train_file_columns = ['click_time','ip','app','device','os','channel','is_attributed'] \n if is_test_file:\n return dtypes, date_columns, test_file_columns\n else:\n return dtypes, date_columns, train_file_columns","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:35.467448Z","iopub.execute_input":"2024-04-19T01:42:35.467751Z","iopub.status.idle":"2024-04-19T01:42:35.475704Z","shell.execute_reply.started":"2024-04-19T01:42:35.467706Z","shell.execute_reply":"2024-04-19T01:42:35.474920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## functions\n\n

    functions for preparing data set

    \n\n\n```python\ndef read_data_file(file_path, is_test_file)\ndef process_ip_bucket(ip_bucket, is_down_sample, majority_multiply, tsfm_sv_policy) #'new_and_save','reuse','new'\n\n# columns created in read_data_file(file_path, is_test_file)\ndf['dd'] = pd.to_datetime(df.click_time).dt.day.astype('uint8') # [01, 31]\ndf['hh'] = pd.to_datetime(df.click_time).dt.hour.astype('uint8') # [00, 23]```","metadata":{}},{"cell_type":"code","source":"def read_data_file(file_path, is_test_file):\n print('read file [is_test_file={}]: {}'.format(is_test_file, file_path))\n dtypes, date_columns, file_columns = get_file_spec(is_test_file)\n df = pd.read_csv(file_path, parse_dates=date_columns,usecols=file_columns,dtype=dtypes)\n df['dd'] = pd.to_datetime(df.click_time).dt.day.astype('uint8') # [01, 31]\n df['hh'] = pd.to_datetime(df.click_time).dt.hour.astype('uint8') # [00, 23]\n g()\n return df\n\ndef process_ip_bucket(ip_bucket, is_down_sample, majority_multiply, tsfm_sv_policy):\n # file path\n print('---------- process bucket: {} -----------'.format(ip_bucket))\n train_file_path = '/kaggle/working/train/train_{}.csv'.format(ip_bucket)\n test_file_path = '/kaggle/working/test/test_{}.csv'.format(ip_bucket)\n # read file\n df_test = read_data_file(test_file_path, is_test_file=True) \n df_train_vldt = read_data_file(train_file_path, is_test_file=False) # df train validation (không downsampled)\n print('read files: {}; {}'.format(train_file_path, test_file_path))\n # align columns and merge\n df_train_vldt['click_id'] = -1\n df_test['is_attributed'] = -1\n df_full = pd.concat([df_train_vldt, df_test], sort=False) # concat test và train của từng IPBucket. VD: df_full_1, df_full_2..., df_full_20\n # train_vldt_len and nunique of raw feature\n train_vldt_len = len(df_train_vldt)\n delete(df_train_vldt, df_test)\n #print('df_full.head(n=5000000).nunique():\\n', df_full.head(n=5000000).nunique()); g()\n #ip:9819; app:343; device:250; os:246; channel:169; click_time:121600; ...\n # feature process\n# print('add features: ')\n# df_full = add_features(df_full, save_transformer=tsfm_sv_policy); g()\n # drop non-training columns\n# print('drop non-training-columns: {}'.format(g_non_train_columns))\n# df_full.drop(g_non_train_columns, axis=1, inplace=True); g()\n # split into df_train, df_vldt, df_test\n print('split data set: ')\n vldt_len = min(g_vldt_set_size // g_ip_bkt_num, train_vldt_len // 5) # min(5tr // 20, valid size // 5)\n df_train = df_full[: train_vldt_len - vldt_len]\n df_vldt = df_full[train_vldt_len - vldt_len : train_vldt_len] \n df_test = df_full[train_vldt_len :]\n # drop tmp colums created when aligning\n print('drop temporary columns: ')\n print('\\tdf_train[\\'click_id\\'].value_counts()={}'.format(df_train['click_id'].value_counts()))\n print('\\tdf_vldt[\\'click_id\\'].value_counts()={}'.format(df_vldt['click_id'].value_counts()))\n print('\\tdf_test[\\'is_attributed\\'].value_counts()={}'.format(df_test['is_attributed'].value_counts()))\n df_train.drop(['click_id'], axis=1, inplace=True)\n df_vldt.drop(['click_id'], axis=1, inplace=True)\n df_test.drop(['is_attributed'], axis=1, inplace=True)\n print('shape: vldt={}; test={}'.format(df_vldt.shape, df_test.shape))\n print('shape: train(before downsample)={}'.format(df_train.shape))\n g()\n # downsample if necessary\n if is_down_sample == True:\n print('down_sample: majority_multiply={}'.format(majority_multiply))\n df_train = random_down_sample(df_train, majority_multiply)\n print('gc.collect (warnning is OK): ')\n g()\n print('shape(after downsample)={}'.format(df_train.shape))\n # return\n return df_train, df_vldt, df_test","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:42:36.979042Z","iopub.execute_input":"2024-04-19T01:42:36.979325Z","iopub.status.idle":"2024-04-19T01:42:36.998689Z","shell.execute_reply.started":"2024-04-19T01:42:36.979284Z","shell.execute_reply":"2024-04-19T01:42:36.997921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Data Set\n\n```python\ndef prep_data_set_full_data()\ndef feature_target_split(df, target_col_name, inplace=False)\ndef prep_feature_target_full_data()\n```","metadata":{}},{"cell_type":"code","source":"def prep_data_set_full_data():\n log_template=\"append bkt {}: train.shape={}; vldt.shape={}; test.shape={}\"\n df_train, df_vldt, df_test = process_ip_bucket(\n ip_bucket=0, is_down_sample= False, \n majority_multiply=g_majority_multiply, \n tsfm_sv_policy='new_and_save')\n print(log_template.format(0, df_train.shape, df_vldt.shape, df_test.shape))\n for bkt_id in range(1,g_ip_bkt_num):\n train_bkt, vldt_bkt, test_bkt = process_ip_bucket(\n ip_bucket=bkt_id, is_down_sample=False, \n majority_multiply=g_majority_multiply, \n tsfm_sv_policy='reuse'); g()\n df_train = df_train.append(train_bkt, ignore_index=True); g()\n df_vldt = df_vldt.append(vldt_bkt, ignore_index=True); g()\n df_test = df_test.append(test_bkt, ignore_index=True); g()\n delete(train_bkt, vldt_bkt, test_bkt)\n print(log_template.format(bkt_id, df_train.shape, df_vldt.shape, df_test.shape))\n df_test.set_index('click_id', drop=True, inplace=True)\n df_test.sort_index(axis=0, inplace=True)\n return df_train, df_vldt, df_test\n\ndef feature_target_split(df, target_col_name, inplace=False):\n y = df[target_col_name]; g()\n if True == inplace:\n df.drop(target_col_name, axis=1, inplace=True); g()\n X = df\n else:\n X = df.drop(target_col_name, axis=1, inplace=True); g()\n return X, y\n\ndef prep_feature_target_full_data(): \n df_train, df_vldt, df_test = prep_data_set_full_data()\n X_train, y_train = feature_target_split(df_train, target_col_name = 'is_attributed', inplace = True)\n X_vldt, y_vldt = feature_target_split(df_vldt, target_col_name = 'is_attributed', inplace = True)\n delete(df_train, df_vldt)\n print('----: prep_feature_target_full_data :-----')\n print('-- X_train={}; y_train={}; X_vldt={}; y_vldt={}; df_test={}'.format(\n X_train.shape, y_train.shape, X_vldt.shape, y_vldt.shape, df_test.shape))\n print('-- features :', X_train.columns.values.tolist())\n print('-- categorical :', g_fit_params['categorical_feature'])\n print('-- y_train.value_counts\\n:', y_train.value_counts())\n print('-- y_vldt.value_countsl\\n:', y_vldt.value_counts())\n return X_train, y_train, X_vldt, y_vldt, df_test","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-19T01:42:38.939677Z","iopub.execute_input":"2024-04-19T01:42:38.940012Z","iopub.status.idle":"2024-04-19T01:42:38.957699Z","shell.execute_reply.started":"2024-04-19T01:42:38.939964Z","shell.execute_reply":"2024-04-19T01:42:38.956747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model and Parameters\n\n## Model Parameters \n\n

    parameters of LGBMClassifier constructor parameter. there are some dicts to manage baseline model and experimental parameters (grid-search) of each baseline

    ","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\nfrom lightgbm import LGBMClassifier\nimport xgboost as xgb\nfrom sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.model_selection import GridSearchCV\n\n# Light Gradient Boosting Classifier Parameters\n# https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html\n# https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html\n# https://lightgbm.readthedocs.io/en/latest/Parameters.html\n\ndef default_model(): \n ''' Function to return LightGBM model with default params'''\n lgb_default = LGBMClassifier()\n return lgb_default.set_params(\n objective = 'binary', # for binary classification\n metric = 'auc', # use metric required by this contest\n boosting_type = 'gbdt', # gbdt, dart, goss, rf\n verbose = 1, # -1\n nthread = 4, \n iid = False, # return average score across folds (not weighted)\n two_round = True \n )\n \ndef gbtd_base_001():\n '''Baseline LightGBM 1'''\n return default_model().set_params(\n subsample = 0.8, # alias of bagging_fraction\n subsample_freq = 1, # alias of bagging_freq\n subsample_for_bin = 200000, # alias of bin_construct_sample_cnt\n colsample_bytree = 0.8, # alias of feature_fraction\n learning_rate = 0.08, \n num_leaves = 105, # 100:v103 -> 105:v105\n max_depth = 7,\n min_split_gain = 0.3, \n max_bin = 255, # default 255\n reg_alpha = 0.3, \n n_estimators = 2500 # alias of num_boost_round\n )\n\ng_search_params = {\n # https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html\n # 'gbdt_base_001_exp_001' : {'num_leaves':[90, 95, 100, 105]}, # 105\n # 'gbdt_base_001_exp_003' : {'learning_rate':[0.1, 0.08, 0.06]}, # 0.8 is the best\n # 'gbdt_base_001_exp_004' : {'max_bin':[255, 315]}, # no difference\n 'gbdt_base_001_exp_002' : {'min_sum_hessian_in_leaf':[0.001, 0.01, 0.05, 0.1]}, \n 'gbdt_base_001_exp_005' : {'min_split_gain':[0.3, 0.4, 0.5]},\n 'gbdt_base_001_exp_010' : {'num_leaves':[90,105], 'min_split_gain':[0.3,0.4]},\n}\n\ng_base_models = {\n 'gbdt_base_001' : gbtd_base_001(),\n}\n\ndef update_data_balancing_param(lgb_model, y_value_counts, majority_val=0, minority_val=1):\n print('y_value_counts: \\n{}'.format(y_value_counts))\n if g_scale_pos_weight is None:\n print(f'use global config: scale_pos_weight={g_scale_pos_weight}')\n lgb_model.set_params(scale_pos_weight = g_scale_pos_weight)\n else :\n unbalance_degree = y_value_counts[majority_val] / y_value_counts[minority_val]\n print(f'majority_count/minority_count={unbalance_degree}') \n if unbalance_degree > 1.5 or unbalance_degree < 0.66:\n print(f'unbalance_dgree')\n lgb_model.set_params(is_unbalance = True)\n return lgb_model\n\ndef get_model_and_search_params(base_model_id, search_params_id):\n return g_base_models[base_model_id], g_search_params[search_params_id]\n\n\n# test\n# model, search_params = get_model_and_search_params('gbdt_base_001', 'gbdt_base_001_exp_001')\n# print(model.get_params(), '\\n', search_params)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:40.462628Z","iopub.execute_input":"2024-04-19T01:42:40.462963Z","iopub.status.idle":"2024-04-19T01:42:40.571725Z","shell.execute_reply.started":"2024-04-19T01:42:40.462904Z","shell.execute_reply":"2024-04-19T01:42:40.570868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fit-Parameters \n\nparameters of LGBMClassifier.fit() function","metadata":{}},{"cell_type":"code","source":"g_fit_params = {\n 'categorical_feature' : g_categorical_features,\n 'early_stopping_rounds' : 25,\n 'verbose' : 10,\n 'eval_metric' : 'auc'\n}","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:41.411390Z","iopub.execute_input":"2024-04-19T01:42:41.411688Z","iopub.status.idle":"2024-04-19T01:42:41.415620Z","shell.execute_reply.started":"2024-04-19T01:42:41.411645Z","shell.execute_reply":"2024-04-19T01:42:41.414888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Function for fitting model","metadata":{}},{"cell_type":"code","source":"def fit_model(X_train, y_train, X_validate, y_validate, lgb_model): \n '''\n id_in_g_base_models: hash key in g_base_models, such as 'gbdt_base_001'\n '''\n # fit parameters\n # tìm index của các category feature\n cat_fea_indices = list(map(lambda col:X_train.columns.get_loc(col), g_fit_params['categorical_feature']))\n print('------ fit parameters: ---------------------')\n print('early_stopping_rounds: {}'.format(g_fit_params['early_stopping_rounds']))\n print('verbose: {}'.format(g_fit_params['verbose']))\n print('eval_metric: {}'.format(g_fit_params['eval_metric']))\n print('categorical_feature: ', cat_fea_indices)\n for i in cat_fea_indices:\n print(\"\\t\", i, \":\", g_fit_params['categorical_feature'][i])\n print('------ update balancing parameters: ---------------------')\n lgb_model = update_data_balancing_param(lgb_model, y_train.value_counts())\n print('------ model parameters: ---------------------') \n print(lgb_model.get_params())\n # fit model\n fitted = lgb_model.fit(X_train, y_train \n , categorical_feature = cat_fea_indices \n , early_stopping_rounds = g_fit_params['early_stopping_rounds']\n , verbose = g_fit_params['verbose']\n , eval_metric = g_fit_params['eval_metric']\n , eval_set = [(X_validate, y_validate)])\n # return\n g()\n return fitted","metadata":{"execution":{"iopub.status.busy":"2024-04-19T01:42:43.700818Z","iopub.execute_input":"2024-04-19T01:42:43.701160Z","iopub.status.idle":"2024-04-19T01:42:43.711607Z","shell.execute_reply.started":"2024-04-19T01:42:43.701113Z","shell.execute_reply":"2024-04-19T01:42:43.710756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fit Model\n\nfetch a baseline model \n\n> 选取一个基线模型","metadata":{}},{"cell_type":"code","source":"g_base_model = g_base_models['gbdt_base_001']","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:35.739462Z","iopub.execute_input":"2024-04-17T09:22:35.739823Z","iopub.status.idle":"2024-04-17T09:22:35.744016Z","shell.execute_reply.started":"2024-04-17T09:22:35.739757Z","shell.execute_reply":"2024-04-17T09:22:35.742876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# g_base_model1 = g_base_models['gbdt_base_002']","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:37.480819Z","iopub.execute_input":"2024-04-17T09:22:37.481160Z","iopub.status.idle":"2024-04-17T09:22:37.485208Z","shell.execute_reply.started":"2024-04-17T09:22:37.481105Z","shell.execute_reply":"2024-04-17T09:22:37.484146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"prepare dataset， including train-set, validate-set, test-set(for submit)","metadata":{}},{"cell_type":"code","source":"# with timer_memory('prep_data_set_full_data()'):\n# df_train, df_vldt, df_test = prep_data_set_full_data()","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:38.775988Z","iopub.execute_input":"2024-04-17T09:22:38.776322Z","iopub.status.idle":"2024-04-17T09:22:38.779854Z","shell.execute_reply.started":"2024-04-17T09:22:38.776264Z","shell.execute_reply":"2024-04-17T09:22:38.778916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_test.to_csv('test_set_after_fe.csv')","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:39.378499Z","iopub.execute_input":"2024-04-17T09:22:39.378817Z","iopub.status.idle":"2024-04-17T09:22:39.382834Z","shell.execute_reply.started":"2024-04-17T09:22:39.378774Z","shell.execute_reply":"2024-04-17T09:22:39.381198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_train.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:39.664387Z","iopub.execute_input":"2024-04-17T09:22:39.664728Z","iopub.status.idle":"2024-04-17T09:22:39.668508Z","shell.execute_reply.started":"2024-04-17T09:22:39.664676Z","shell.execute_reply":"2024-04-17T09:22:39.667568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('prep_feature_target_full_data'):\n# X_train, y_train, g_X_vldt, g_y_vldt, g_df_test = prep_feature_target_full_data()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-17T09:22:40.240880Z","iopub.execute_input":"2024-04-17T09:22:40.241296Z","iopub.status.idle":"2024-04-17T09:22:40.247743Z","shell.execute_reply.started":"2024-04-17T09:22:40.241232Z","shell.execute_reply":"2024-04-17T09:22:40.246661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Slit into 5 sub-samples and down sampling each sub-sample","metadata":{}},{"cell_type":"code","source":"# # Count negative samples\n# num_negative_samples = (df_train['is_attributed'] == 0).sum()\n\n# # Sample only 5% of negative samples\n# sampled_negative_indices = df_train[df_train['is_attributed'] == 0].sample(frac=0.05).index\n\n# # Keep all positive samples\n# positive_samples = df_train[df_train['is_attributed'] == 1]\n\n# # Keep sampled negative samples\n# sampled_negative_samples = df_train.loc[sampled_negative_indices]\n\n# # Concatenate positive and sampled negative samples\n# reduced_df_train = pd.concat([positive_samples, sampled_negative_samples])\n\n# # Shuffle the data\n# reduced_df_train = reduced_df_train.sample(frac=1).reset_index(drop=True)\n\n# # Now reduced_data contains the dataset with only 5% of negative samples retained","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:41.230005Z","iopub.execute_input":"2024-04-17T09:22:41.230305Z","iopub.status.idle":"2024-04-17T09:22:41.233917Z","shell.execute_reply.started":"2024-04-17T09:22:41.230263Z","shell.execute_reply":"2024-04-17T09:22:41.233126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Feature engineering (v1)\n\n# print('add features: ')\n# reduced_df_train = add_features(reduced_df_train, save_transformer='reuse'); g() \n# # drop non-training columns\n# print('drop non-training-columns: {}'.format(g_non_train_columns))\n# reduced_df_train.drop(g_non_train_columns, axis=1, inplace=True); g()","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:41.777099Z","iopub.execute_input":"2024-04-17T09:22:41.777413Z","iopub.status.idle":"2024-04-17T09:22:41.781229Z","shell.execute_reply.started":"2024-04-17T09:22:41.777371Z","shell.execute_reply":"2024-04-17T09:22:41.780227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Calculate the number of samples for class 0 for each sampled dataset as a proportion\n# class_0_instances = reduced_df_train[reduced_df_train['is_attributed'] == 0]\n# num_samples_class_0 = int(len(class_0_instances))","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:42.169144Z","iopub.execute_input":"2024-04-17T09:22:42.169480Z","iopub.status.idle":"2024-04-17T09:22:42.172919Z","shell.execute_reply.started":"2024-04-17T09:22:42.169412Z","shell.execute_reply":"2024-04-17T09:22:42.172051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Calculate the number of samples for class 0 for each sampled dataset\n# num_samples_class_0_per_dataset = num_samples_class_0 // 7 # Assuming equal division\n\n# # Randomly sample class 0 data for each sampled dataset using different random seeds\n# sampled_datasets = []\n# for seed in [123, 456, 789, 321, 654, 987, 512]: # Different random seeds\n# np.random.seed(seed)\n \n# # Sample class 0 data\n# sampled_class_0 = reduced_df_train[reduced_df_train['is_attributed'] == 0].sample(n=num_samples_class_0_per_dataset, replace=False)\n \n# # Include all instances of class 1\n# sampled_class_1 = reduced_df_train[reduced_df_train['is_attributed'] == 1]\n \n# # Combine class 0 and class 1 samples\n# sampled_dataset = pd.concat([sampled_class_0, sampled_class_1])\n \n# # Shuffle the dataset\n# sampled_dataset = sampled_dataset.sample(frac=1).reset_index(drop=True)\n# # sampled_dataset.to_csv(f'dataset_{seed}.csv')\n \n# # Append to the list of sampled datasets\n# sampled_datasets.append(sampled_dataset)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:42.659799Z","iopub.execute_input":"2024-04-17T09:22:42.660231Z","iopub.status.idle":"2024-04-17T09:22:42.664593Z","shell.execute_reply.started":"2024-04-17T09:22:42.660152Z","shell.execute_reply":"2024-04-17T09:22:42.663685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Downsample each dataset \n# # downsampled_datasets = []\n# for i, sample in enumerate(sampled_datasets):\n# s = random_down_sample(sample)\n# # downsampled_datasets.append(s)\n# s.to_csv(f'dataset_{i}.csv')","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:43.208516Z","iopub.execute_input":"2024-04-17T09:22:43.208853Z","iopub.status.idle":"2024-04-17T09:22:43.212666Z","shell.execute_reply.started":"2024-04-17T09:22:43.208805Z","shell.execute_reply":"2024-04-17T09:22:43.211577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# downsampled_datasets[0].columns","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:43.631686Z","iopub.execute_input":"2024-04-17T09:22:43.631986Z","iopub.status.idle":"2024-04-17T09:22:43.635391Z","shell.execute_reply.started":"2024-04-17T09:22:43.631945Z","shell.execute_reply":"2024-04-17T09:22:43.634572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print('add features: ')\n# df_vldt = add_features(df_vldt, save_transformer='reuse'); g() \n# # drop non-training columns\n# print('drop non-training-columns: {}'.format(g_non_train_columns))\n# df_vldt.drop(g_non_train_columns, axis=1, inplace=True); g()\n# df_vldt.to_csv('vldt_set_after_fe.csv')","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:44.081857Z","iopub.execute_input":"2024-04-17T09:22:44.082196Z","iopub.status.idle":"2024-04-17T09:22:44.085882Z","shell.execute_reply.started":"2024-04-17T09:22:44.082139Z","shell.execute_reply":"2024-04-17T09:22:44.084954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_vldt.shape","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:22:44.393365Z","iopub.execute_input":"2024-04-17T09:22:44.393677Z","iopub.status.idle":"2024-04-17T09:22:44.396867Z","shell.execute_reply.started":"2024-04-17T09:22:44.393628Z","shell.execute_reply":"2024-04-17T09:22:44.396030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_vldt = pd.read_csv('/kaggle/input/competition-talkingdata-adtracking-fraudde-2/vldt_set_after_fe.csv')\nX_vldt = df_vldt.drop(['is_attributed', 'short_text', df_vldt.columns[0]], axis = 1)\ny_vldt = df_vldt['is_attributed']","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:23:55.619746Z","iopub.execute_input":"2024-04-17T09:23:55.620099Z","iopub.status.idle":"2024-04-17T09:24:26.956341Z","shell.execute_reply.started":"2024-04-17T09:23:55.620041Z","shell.execute_reply":"2024-04-17T09:24:26.955505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trains = []\nROOT = '/kaggle/input/competition-talkingdata-adtracking-fraudde-2/'\n\nfor d in os.listdir('/kaggle/input/competition-talkingdata-adtracking-fraudde-2'):\n if 'dataset_'in d:\n train = ROOT + d\n trains.append(train)\n \ntrains","metadata":{"execution":{"iopub.status.busy":"2024-04-19T02:18:00.369147Z","iopub.execute_input":"2024-04-19T02:18:00.369478Z","iopub.status.idle":"2024-04-19T02:18:00.384567Z","shell.execute_reply.started":"2024-04-19T02:18:00.369415Z","shell.execute_reply":"2024-04-19T02:18:00.383594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_trains = []\ny_trains = []\n\nfor d_train in trains:\n sample = pd.read_csv(d_train)\n X_train = sample.drop(['is_attributed', 'short_text', sample.columns[0]], axis=1)\n y_train = sample['is_attributed']\n X_trains.append(X_train)\n y_trains.append(y_train)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:24:26.974525Z","iopub.execute_input":"2024-04-17T09:24:26.974797Z","iopub.status.idle":"2024-04-17T09:25:06.817246Z","shell.execute_reply.started":"2024-04-17T09:24:26.974749Z","shell.execute_reply":"2024-04-17T09:25:06.816489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sample_1 = pd.read_csv('/kaggle/working/dataset_123.csv')\n# X_train = sample_1.drop(['is_attributed', X_train.columns[0]], axis = 1)\n# y_train = sample_1['is_attributed']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"fit model","metadata":{}},{"cell_type":"code","source":"# X_trains[0].columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_trains[0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# X_vldt","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Define a list to store fitted models\n# fitted_models = []\n\n# # Iterate through the five sampled datasets\n# for i in range(len(X_trains)):\n \n# # print(X_train)\n# # print(y_train)\n# # break\n# # Fit the model for the current sampled dataset\n# with timer_memory(f'fit_model_{i}'):\n# g_model_fitted = fit_model(X_trains[i], y_trains[i], X_vldt, y_vldt, g_base_model)\n \n# g_model_fitted.booster_.save_model(f'/kaggle/working/mode_{i + 5}.txt')\n# fitted_models.append(g_model_fitted)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_1 = fit_model(X_trains[0], y_trains[0], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-04-17T09:26:57.460742Z","iopub.execute_input":"2024-04-17T09:26:57.461050Z","iopub.status.idle":"2024-04-17T09:27:20.501466Z","shell.execute_reply.started":"2024-04-17T09:26:57.461009Z","shell.execute_reply":"2024-04-17T09:27:20.499666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_2 = fit_model(X_trains[1], y_trains[1], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:27:27.994189Z","iopub.execute_input":"2024-04-17T09:27:27.994488Z","iopub.status.idle":"2024-04-17T09:27:50.797438Z","shell.execute_reply.started":"2024-04-17T09:27:27.994445Z","shell.execute_reply":"2024-04-17T09:27:50.796570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_3 = fit_model(X_trains[2], y_trains[2], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:27:50.800065Z","iopub.execute_input":"2024-04-17T09:27:50.800389Z","iopub.status.idle":"2024-04-17T09:28:14.170762Z","shell.execute_reply.started":"2024-04-17T09:27:50.800332Z","shell.execute_reply":"2024-04-17T09:28:14.169875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_4 = fit_model(X_trains[3], y_trains[3], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:28:14.172205Z","iopub.execute_input":"2024-04-17T09:28:14.172571Z","iopub.status.idle":"2024-04-17T09:32:16.665368Z","shell.execute_reply.started":"2024-04-17T09:28:14.172496Z","shell.execute_reply":"2024-04-17T09:32:16.664593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_5 = fit_model(X_trains[4], y_trains[4], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:32:16.666941Z","iopub.execute_input":"2024-04-17T09:32:16.667261Z","iopub.status.idle":"2024-04-17T09:35:16.674679Z","shell.execute_reply.started":"2024-04-17T09:32:16.667200Z","shell.execute_reply":"2024-04-17T09:35:16.673838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_6 = fit_model(X_trains[5], y_trains[5], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:35:16.678059Z","iopub.execute_input":"2024-04-17T09:35:16.678347Z","iopub.status.idle":"2024-04-17T09:35:40.044103Z","shell.execute_reply.started":"2024-04-17T09:35:16.678304Z","shell.execute_reply":"2024-04-17T09:35:40.043097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('fit_model'):\n g_model_fitted_7 = fit_model(X_trains[6], y_trains[6], X_vldt, y_vldt, g_base_model)\n# g_model_fitted = fit_model(X_train, y_train, g_X_vldt, g_y_vldt, g_base_model1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:35:40.045693Z","iopub.execute_input":"2024-04-17T09:35:40.045946Z","iopub.status.idle":"2024-04-17T09:36:03.121593Z","shell.execute_reply.started":"2024-04-17T09:35:40.045900Z","shell.execute_reply":"2024-04-17T09:36:03.120735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict and Submit\n\npredict on test-set and generate submission file\n","metadata":{}},{"cell_type":"code","source":"fitted_models = [g_model_fitted_1, g_model_fitted_2, g_model_fitted_3, g_model_fitted_4, g_model_fitted_5, g_model_fitted_6, g_model_fitted_7]","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:36:03.122873Z","iopub.execute_input":"2024-04-17T09:36:03.123114Z","iopub.status.idle":"2024-04-17T09:36:03.126778Z","shell.execute_reply.started":"2024-04-17T09:36:03.123075Z","shell.execute_reply":"2024-04-17T09:36:03.125993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# def predict_and_submit(model_fitted, num_iteration):\n# sub = pd.DataFrame()\n# sub['click_id'] = df_test.index\n# sub['click_id'] = sub['click_id'].astype('int')\n# pred_prob = model_fitted.predict_proba(X=df_test, num_iteration=num_iteration)\n# sub['is_attributed'] = pred_prob[:,1].reshape(-1,1)\n# print(f'num_iteration: {num_iteration}')\n# print(f'sub.shape: {sub.shape}')\n# return sub\n\ndef predict_and_submit(i, model_fitted, num_iteration):\n sub = pd.DataFrame()\n sub['click_id'] = df_test.index\n sub['click_id'] = sub['click_id'].astype('int')\n pred_prob = model_fitted.predict_proba(X=df_test, num_iteration=num_iteration)\n sub['is_attributed'] = pred_prob[:,1].reshape(-1,1)\n sub.to_csv(f'submit{i}.csv', index=False, float_format='%.9f')\n print(f'num_iteration: {num_iteration}')\n print(f'sub.shape: {sub.shape}')\n return sub","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:36:03.127933Z","iopub.execute_input":"2024-04-17T09:36:03.128147Z","iopub.status.idle":"2024-04-17T09:36:03.140801Z","shell.execute_reply.started":"2024-04-17T09:36:03.128111Z","shell.execute_reply":"2024-04-17T09:36:03.140122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = pd.read_csv('/kaggle/input/fe-test-competition-talkingdata-adtracking-fraudd/test_set_after_fe.csv')","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:50:07.681023Z","iopub.execute_input":"2024-04-17T09:50:07.681336Z","iopub.status.idle":"2024-04-17T09:51:45.855253Z","shell.execute_reply.started":"2024-04-17T09:50:07.681295Z","shell.execute_reply":"2024-04-17T09:51:45.854265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:52:28.727165Z","iopub.execute_input":"2024-04-17T09:52:28.727467Z","iopub.status.idle":"2024-04-17T09:52:37.921997Z","shell.execute_reply.started":"2024-04-17T09:52:28.727419Z","shell.execute_reply":"2024-04-17T09:52:37.921146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test = df_test.drop(['short_text', df_test.columns[0]], axis = 1)\ndf_test","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:52:54.555790Z","iopub.execute_input":"2024-04-17T09:52:54.556279Z","iopub.status.idle":"2024-04-17T09:53:04.213259Z","shell.execute_reply.started":"2024-04-17T09:52:54.556064Z","shell.execute_reply":"2024-04-17T09:53:04.212363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_vldt","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:53:38.460565Z","iopub.execute_input":"2024-04-17T09:53:38.460895Z","iopub.status.idle":"2024-04-17T09:53:39.930260Z","shell.execute_reply.started":"2024-04-17T09:53:38.460854Z","shell.execute_reply":"2024-04-17T09:53:39.929502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submits = []","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:36:03.141966Z","iopub.execute_input":"2024-04-17T09:36:03.142211Z","iopub.status.idle":"2024-04-17T09:36:03.150955Z","shell.execute_reply.started":"2024-04-17T09:36:03.142165Z","shell.execute_reply":"2024-04-17T09:36:03.150213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit1 = predict_and_submit(1, g_model_fitted_1, g_model_fitted_1.best_iteration_)\n# submit1.head()\n# submits.append(submit1)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:53:55.669470Z","iopub.execute_input":"2024-04-17T09:53:55.669807Z","iopub.status.idle":"2024-04-17T09:56:05.267552Z","shell.execute_reply.started":"2024-04-17T09:53:55.669759Z","shell.execute_reply":"2024-04-17T09:56:05.266734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit2 = predict_and_submit(2, g_model_fitted_2, g_model_fitted_2.best_iteration_)\n# submit2.head()\n# submits.append(submit2)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:57:29.755019Z","iopub.execute_input":"2024-04-17T09:57:29.755357Z","iopub.status.idle":"2024-04-17T09:59:35.900088Z","shell.execute_reply.started":"2024-04-17T09:57:29.755305Z","shell.execute_reply":"2024-04-17T09:59:35.899275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit3 = predict_and_submit(3, g_model_fitted_3, g_model_fitted_3.best_iteration_)\n# submit3.head()\n# submits.append(submit3)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:59:40.241675Z","iopub.execute_input":"2024-04-17T09:59:40.241988Z","iopub.status.idle":"2024-04-17T10:01:48.302746Z","shell.execute_reply.started":"2024-04-17T09:59:40.241941Z","shell.execute_reply":"2024-04-17T10:01:48.301762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit4 = predict_and_submit(4, g_model_fitted_4, g_model_fitted_4.best_iteration_)\n# submit4.head()\n# submits.append(submit4)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T10:01:48.304643Z","iopub.execute_input":"2024-04-17T10:01:48.304968Z","iopub.status.idle":"2024-04-17T10:03:54.993771Z","shell.execute_reply.started":"2024-04-17T10:01:48.304913Z","shell.execute_reply":"2024-04-17T10:03:54.992757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit5 = predict_and_submit(5, g_model_fitted_5, g_model_fitted_5.best_iteration_)\n# submit5.head()\n# submits.append(submit5)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T10:03:54.995933Z","iopub.execute_input":"2024-04-17T10:03:54.996260Z","iopub.status.idle":"2024-04-17T10:06:01.374407Z","shell.execute_reply.started":"2024-04-17T10:03:54.996201Z","shell.execute_reply":"2024-04-17T10:06:01.373638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit6 = predict_and_submit(6, g_model_fitted_6, g_model_fitted_6.best_iteration_)\n# submit6.head()\n# submits.append(submit6)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T10:06:01.375867Z","iopub.execute_input":"2024-04-17T10:06:01.376109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with timer_memory('predict_and_sumit'):\n submit7 = predict_and_submit(7, g_model_fitted_7, g_model_fitted_7.best_iteration_)\n submit7.head()\n submits.append(submit7)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Initialize an empty DataFrame to store 'is_attributed' columns from all submit files\n# submit_data = pd.DataFrame()\n\n# # Loop through each submit file\n# for sub in submits:\n# # Collect the 'is_attributed' column and append it to the submit_data DataFrame\n# submit_data[f'submit{i}'] = sub['is_attributed']\n\n# # Compute the mean of each row across all submit files\n# submit_data['mean_is_attributed'] = submit_data.mean(axis=1)\n\n\n# # Replace the 'is_attributed' column in the original submit file with the computed mean\n# submit1['is_attributed'] = submit_data['mean_is_attributed']\n\n# # Save the modified original submit file\n# submit1.to_csv('submit.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:36:03.190633Z","iopub.status.idle":"2024-04-17T09:36:03.191211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submit1","metadata":{"execution":{"iopub.status.busy":"2024-04-17T09:36:03.192419Z","iopub.status.idle":"2024-04-17T09:36:03.192970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Directory containing the files\n# directory_path = '/kaggle/input/demo-test'\n\n# # List to store DataFrames from each file\n# dataframes = []\n\n# # Loop through each file in the directory\n# for filename in os.listdir(directory_path):\n# filepath = os.path.join(directory_path, filename)\n# if os.path.isfile(filepath):\n# # Read the file into a DataFrame\n# df = pd.read_csv(filepath)\n# # Append DataFrame to the list\n# dataframes.append(df)\n\n# # Concatenate all DataFrames into one\n# merged_df = pd.concat(dataframes, ignore_index=True)\n# merged_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# df_test = df_test.drop(['click_id','short_text', merged_df.columns[0]], axis = 1)\n# df_test","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit = predict_and_submit(g_model_fitted, g_model_fitted.best_iteration_)\n# submit.head()\n# submit.to_csv(f'submit5.csv', index=False, float_format='%.9f')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit = predict_and_submit(fitted_models[1], fitted_models[1].best_iteration_)\n# submit.head()\n# submit.to_csv(f'submit6.csv', index=False, float_format='%.9f')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit = predict_and_submit(fitted_models[2], fitted_models[2].best_iteration_)\n# submit.head()\n# submit.to_csv(f'submit5.csv', index=False, float_format='%.9f')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit = predict_and_submit(fitted_models[3], fitted_models[3].best_iteration_)\n# submit.head()\n# submit.to_csv(f'submit6.csv', index=False, float_format='%.9f')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Initialize an empty DataFrame to store 'is_attributed' columns from all submit files\n# submit_data = pd.DataFrame()\n\n# # Loop through each submit file\n# for i in range(7):\n# # Load the submit file\n# submit_file_path = f\"/kaggle/working/submit{i}.csv\"\n# submit_df = pd.read_csv(submit_file_path)\n \n# # Collect the 'is_attributed' column and append it to the submit_data DataFrame\n# submit_data[f'submit{i}'] = submit_df['is_attributed']\n\n# # Compute the mean of each row across all submit files\n# submit_data['mean_is_attributed'] = submit_data.mean(axis=1)\n\n# # Choose one original submit file to replace its 'is_attributed' column with the computed mean\n# original_submit_file_path = \"/kaggle/working/submit0.csv\" # Choose any one original submit file\n# original_submit_df = pd.read_csv(original_submit_file_path)\n\n# # Replace the 'is_attributed' column in the original submit file with the computed mean\n# original_submit_df['is_attributed'] = submit_data['mean_is_attributed']\n\n# # Save the modified original submit file\n# original_submit_df.to_csv('submit.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# os.chdir('/kaggle/working')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from IPython.display import FileLink\n# FileLink(r'submit.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# with timer_memory('predict_and_sumit'):\n# submit = predict_and_submit(g_model_fitted, g_model_fitted.best_iteration_)\n# submit.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate Model\n\nfunctions for evaluating model\n\n```python\ndef plot_roc_auc_curve(y_label, y_proba_pred)\ndef plot_pr_curve(y_label, y_proba_pred)\ndef plot_confusion_matrix(y_label, y_label_pred)```","metadata":{}},{"cell_type":"code","source":"def plot_roc_auc_curve(y_label, y_proba_pred): \n from sklearn.metrics import roc_auc_score\n from sklearn.metrics import roc_curve\n import matplotlib.pyplot as plt \n auc_score = roc_auc_score(y_label.values, y_proba_pred)\n fpr, tpr, thresh = roc_curve(y_label.values, y_proba_pred, pos_label=None)\n plt.figure(figsize=(6,6))\n plt.title('ROC curve')\n plt.plot(fpr, tpr, label='auc score: {:.4f}'.format(auc_score))\n plt.plot([0, 1], [0, 1], 'k--')\n plt.axis([-0.01, 1, 0, 1])\n plt.xlabel('False Positive Rate')\n plt.ylabel('True Positive Rate')\n plt.annotate('Minimum ROC Score of 50% \\n (This is the minimum score to get)'\n ,xy=(0.5, 0.5), xytext=(0.6, 0.3), arrowprops=dict(facecolor='#6E726D', shrink=0.05))\n plt.legend()\n plt.show()\n\ndef plot_pr_curve(y_label, y_proba_pred):\n import matplotlib.pyplot as plt\n from sklearn.metrics import precision_recall_curve\n precision, recall, threshold = precision_recall_curve(y_label.values, y_proba_pred)\n plt.figure(figsize=(6,6))\n plt.title('precision-recall curve', fontsize=16)\n plt.plot(recall, precision, 'b-', linewidth=2)\n plt.plot([0, 1], [0, 1], 'r--')\n plt.xlabel('precision')\n plt.ylabel('recall')\n plt.axis([-0.01,1,0,1])\n plt.show()\n\ndef plot_confusion_matrix(y_label, y_label_pred):\n import matplotlib.pyplot as plt\n from sklearn.metrics import confusion_matrix\n cfsn_matrix = confusion_matrix(y_label.values, y_label_pred)\n fig, ax = plt.subplots(1, 1,figsize=(6,6))\n sns.heatmap(cfsn_matrix, annot=True, cmap=plt.cm.copper)\n ax.set_title(\"confusion matrix\")\n ax.set_xticklabels(['0', '1'], rotation=90)\n ax.set_yticklabels(['0', '1'], rotation=360)\n ax.set_xlabel('predict labels')\n ax.set_ylabel('true labels')\n plt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"in order to reflect the prediction ability in production environment, label distribution must be the same as in origional data-set (no downsampling)\n","metadata":{}},{"cell_type":"code","source":"# # data-set must be original un-balanced data, can not be under-sampled\n# g_y_pred_proba_vldt = (fitted_models[0].predict_proba(X=X_vldt, num_iteration=g_model_fitted.best_iteration_)[:,1] >= 0.5).astype(int)\n# g_y_pred_label_vldt = g_model_fitted.predict(X=X_vldt, num_iteration=g_model_fitted.best_iteration_)\n# y_vldt.value_counts()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import numpy as np\n\n# # Initialize an array to store predicted probabilities from each model\n# all_pred_proba = []\n\n# # Loop through each fitted model\n# for model in fitted_models:\n# # Predict probabilities for the positive class (1)\n# y_pred_proba = model.predict_proba(X=X_vldt, num_iteration=model.best_iteration_)[:, 1]\n \n# # Append predicted probabilities to the array\n# all_pred_proba.append(y_pred_proba)\n\n# # Convert the array to a NumPy array\n# all_pred_proba = np.array(all_pred_proba)\n\n# # Compute the average predicted probability across all models\n# avg_pred_proba = np.mean(all_pred_proba, axis=0)\n\n# # Threshold the average predicted probability to obtain predicted labels\n# avg_pred_label = (avg_pred_proba >= 0.5).astype(int)\n\n# # Evaluate the performance metrics\n# # Example: You can use metrics like accuracy, precision, recall, etc.\n# from sklearn.metrics import accuracy_score, precision_score, recall_score\n\n# accuracy = accuracy_score(y_vldt, avg_pred_label)\n# precision = precision_score(y_vldt, avg_pred_label)\n# recall = recall_score(y_vldt, avg_pred_label)\n\n# print(\"Average Predictions:\")\n# print(f\"Accuracy: {accuracy:.4f}, Precision: {precision:.4f}, Recall: {recall:.4f}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"evaluate models","metadata":{}},{"cell_type":"code","source":"# plot_roc_auc_curve(y_vldt, avg_pred_label)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot_roc_auc_curve(y_vldt, g_y_pred_proba_vldt)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot_pr_curve(y_vldt, avg_pred_label)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot_pr_curve(y_vldt, g_y_pred_proba_vldt)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from sklearn.metrics import classification_report\n# report = classification_report(y_vldt, avg_pred_label, target_names=['is_not_attributed','is_attributed'])\n# print(report)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from sklearn.metrics import classification_report\n# report = classification_report(y_vldt, g_y_pred_label_vldt, target_names=['is_not_attributed','is_attributed'])\n# print(report)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot_confusion_matrix(y_vldt, g_y_pred_label_vldt)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Features\n\nfunctions for plotting features\n\n\n```python\ndef plot_num_fea_distribution(df, categorical=g_categorical_features)\ndef headmap_plot_fea_target_corr_matrix(df, categorical = g_categorical_features)\ndef boxplot_of_fea_target_corr(df, categorical = g_categorical_features)```","metadata":{}},{"cell_type":"code","source":"def headmap_plot_fea_target_corr_matrix(df, categorical = g_categorical_features):\n corr = df.drop(g_categorical_features, axis=1).corr()\n ax = sns.heatmap(corr, cmap='coolwarm_r', annot_kws={'size':20})\n ax.set_title('Correlation Matrix \\n', fontsize=14)\n plt.show(); g()\n\ndef plot_num_fea_distribution(df, categorical = g_categorical_features):\n feature_names = df.columns.values.tolist()\n for cat_fea_name in categorical + ['is_attributed']:\n if cat_fea_name in feature_names:\n feature_names.remove(cat_fea_name)\n print(\"numerical features: {}\".format(feature_names))\n fig, axs = plt.subplots(ncols=2, nrows=0, figsize=(12, 36))\n plt.subplots_adjust(right=2)\n plt.subplots_adjust(top=2)\n sns.color_palette(\"husl\", 8)\n sns.set_style(\"white\")\n sns.set_color_codes(palette='deep')\n for i, fea_name in enumerate(list(df[feature_names]), 1):\n plt.subplot(len(list(feature_names)), 2, i)\n sns.distplot(df[fea_name], fit=norm, color=\"b\"); \n (mu, sigma) = norm.fit(df[fea_name]) \n plt.legend([\n 'Normal dist. ($\\mu=$ {:.2f} and $\\sigma=$ {:.2f} )'.format(mu, sigma)\n ],loc='best') \n plt.xlabel('{}'.format(fea_name), size=15, labelpad=12.5)\n plt.ylabel(\"Frequency\", size=15, labelpad=12.5)\n for j in range(2):\n plt.tick_params(axis='x', labelsize=12)\n plt.tick_params(axis='y', labelsize=12)\n sns.despine(trim=True, left=True)\n plt.show(); g()\n\ndef boxplot_of_fea_target_corr(df, categorical = g_categorical_features):\n feature_names = df.columns.values.tolist()\n if 'is_attributed' in feature_names:\n feature_names.remove('is_attributed')\n print(\"feature_count: {}\".format(len(feature_names))) \n fig, axs = plt.subplots(ncols=2, nrows=0, figsize=(12, 36))\n plt.subplots_adjust(right=2)\n plt.subplots_adjust(top=2)\n sns.color_palette(\"husl\", 8)\n for i, fea_name in enumerate(list(df[feature_names]), 1):\n if fea_name.startswith(\"cumcount_\"):\n outlier_plot_threshold = 18\n df.loc[df[fea_name]>outlier_plot_threshold, (fea_name)] = outlier_plot_threshold\n plt.subplot(len(list(feature_names)), 2, i)\n if fea_name in categorical:\n #sns.countplot(x=fea_name, hue='is_attributed', data=df)\n sns.violinplot(y=fea_name, x='is_attributed', data=df)\n else:\n sns.boxplot(y=fea_name, x='is_attributed', data=df)\n plt.xlabel('{}'.format('is_attributed'), size=15, labelpad=12.5)\n plt.ylabel(fea_name, size=15, labelpad=12.5)\n for j in range(2):\n plt.tick_params(axis='x', labelsize=12)\n plt.tick_params(axis='y', labelsize=12)\n plt.legend(loc='best', prop={'size': 10})\n plt.show(); g()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"plot feature importance","metadata":{}},{"cell_type":"code","source":"# lgb.plot_importance(g_model_fitted.booster_, importance_type='split')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

    need a down-sampled data set to plot features, this has 2 benifits:

    \n1. virtualizing effect of both majority examples and minority examples: if no down-sampling, almost all input to plotting functions will be majority examples\n2. it is more close with the model training input, which is also down-sampled","metadata":{}},{"cell_type":"code","source":"# df_plot = random_down_sample(\n# X_vldt.merge(y_vldt, left_index=True, right_index=True), \n# majority_multiply=g_majority_multiply, target_col_name='is_attributed', \n# minority_val = 1, majority_val = 0); g()\n# df_plot['is_attributed'].value_counts()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"plot features, including feature distribution, ","metadata":{}},{"cell_type":"code","source":"# plot_num_fea_distribution(df_plot)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# headmap_plot_fea_target_corr_matrix(df_plot)a","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# boxplot_of_fea_target_corr(df_plot)\n# delete(df_plot)","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Analysis","metadata":{}},{"cell_type":"code","source":"# with timer_memory('plot_metric_auc'):\n# lgb.plot_metric(g_model_fitted, 'auc')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"```python\ndef plot_learning_curve(model, X, y, cv, ylim=None, n_jobs=-1, train_sizes=np.array([0.1,0.5,0.75,1])):```","metadata":{}},{"cell_type":"code","source":"# def plot_learning_curve(model, X, y, cv, ylim=None, n_jobs=-1, train_sizes=np.array([0.1,0.5,0.75,1])):\n# import matplotlib.pyplot as plt\n# from sklearn.model_selection import learning_curve\n# fig, ax = plt.subplots(1, 1,figsize=(10,6))\n# if ylim is not None:\n# plt.ylim(*ylim)\n# train_sizes, train_scores, test_scores = learning_curve(\n# model, X, y, cv=cv, shuffle=False, n_jobs=n_jobs, train_sizes=train_sizes)\n# train_scores_mean = np.mean(train_scores, axis=1)\n# train_scores_std = np.std(train_scores, axis=1)\n# test_scores_mean = np.mean(test_scores, axis=1)\n# test_scores_std = np.std(test_scores, axis=1)\n# ax.fill_between(train_sizes, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.1, color=\"#ff9124\")\n# ax.fill_between(train_sizes, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.1, color=\"#2492ff\")\n# ax.plot(train_sizes, train_scores_mean, 'o-', color=\"#ff9124\", label=\"Training score\")\n# ax.plot(train_sizes, test_scores_mean, 'o-', color=\"#2492ff\", label=\"Cross-validation score\")\n# ax.set_title(\"Learning Curve\", fontsize=14)\n# ax.set_xlabel('Training size (m)')\n# ax.set_ylabel('Score')\n# ax.grid(True)\n# ax.legend(loc=\"best\")\n# plt.show()\n\n# from sklearn.model_selection import PredefinedSplit\n# def get_cv_splitter(y_train:pd.Series):\n# total_size, test_size = y_train.size, y_train.size // 10\n# test_fold = np.full(shape=(total_size,), fill_value=-1, dtype='int')\n# test_fold[-1*test_size:] = 0\n# cv = PredefinedSplit(test_fold=test_fold)\n# print(f'total_size={total_size}; test_size={test_size}')\n# return cv","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#with timer_memory('plot_learning_curve'):\n# plot_learning_curve(g_base_model, X=X_train, y=y_train, cv=get_cv_splitter(y_train), ylim=(0.92, 1.01), scoring='roc_auc')","metadata":{"_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Grid Search\n\n

    existing function used:

    \n\n```python\ng_base_models = {\n 'gbdt_base_001' : gbtd_base_001(), ...\n}\ng_search_params = {\n # 'gbdt_base_001_exp_001' : {'num_leaves':[90, 95, 100, 105]}, # 105\n # 'gbdt_base_001_exp_003' : {'learning_rate':[0.1, 0.08, 0.06]}, # 0.8 is the best\n # 'gbdt_base_001_exp_004' : {'max_bin':[255, 315]}, # no difference\n 'gbdt_base_001_exp_002' : {'min_sum_hessian_in_leaf':[0.001, 0.01, 0.05, 0.1]}, \n 'gbdt_base_001_exp_005' : {'min_split_gain':[0.3, 0.4, 0.5]},\n 'gbdt_base_001_exp_010' : {'num_leaves':[90,105], 'min_split_gain':[0.3,0.4]}\n}\ndef get_model_and_search_params(base_model_id, search_params_id)\ndef update_data_balancing_param(lgb_model, y_value_counts, majority_val=0, minority_val=1)\n```\n\n

    new function defined as belows:

    \n```python\ndef grid_search(X_train, y_train, X_vldt, y_vldt, base_model, param_grid)```\n

    parameters turning:

    \n* [https://lightgbm.readthedocs.io/en/latest/Features.html#leaf-wise-best-first-tree-growth](https://lightgbm.readthedocs.io/en/latest/Features.html#leaf-wise-best-first-tree-growth)\n* [https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html](https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html)\n","metadata":{}},{"cell_type":"code","source":"# import lightgbm as lgb\n# import xgboost as xgb\n# from sklearn.model_selection import ParameterGrid\n# from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score, accuracy_score, classification_report\n\n# def grid_search(X_train, y_train, X_vldt, y_vldt, base_model, param_grid):\n# # result\n# estimator_lst,grid_point_lst,precision_lst,recall_lst,f1_lst,auc_lst=[],[],[],[],[],[]\n# # grid search\n# for grid_point in list(ParameterGrid(param_grid)):\n# # model parameters\n# print(f'----- grid_point: \\n{grid_point}')\n# base_model = base_model.set_params(**grid_point)\n# base_model = base_model.set_params(silent = True)\n# base_model = base_model.set_params(verbosity = 0)\n# print('update_data_balancing_param:' )\n# base_model = update_data_balancing_param(base_model, y_train.value_counts())\n# print(f'model parameters: \\n{base_model.get_params()}')\n# # fit parameters\n# print('cat_fea: {}'.format(g_fit_params['categorical_feature']))\n# print('fea: {}'.format(X_train.columns))\n# print(type(X_train.columns))\n# print(X_train.columns.get_loc('app'))\n# cat_fea_indices = list(map(lambda col:X_train.columns.get_loc(col), g_fit_params['categorical_feature']))\n# fit_params_copy = g_fit_params.copy()\n# fit_params_copy['categorical_feature'] = cat_fea_indices\n# fit_params_copy['verbose'] = -1\n# print(f'fit_params_copy: \\n{fit_params_copy}')\n# # fit\n# fitted = base_model.fit(\n# X_train, y_train, eval_set = [(X_vldt, y_vldt)], **fit_params_copy); g()\n# # pred\n# y_pred = fitted.predict(X_vldt)\n# # append_score\n# grid_point_lst.append(grid_point) \n# estimator_lst.append(fitted)\n# precision_lst.append(precision_score(y_vldt, y_pred))\n# recall_lst.append(recall_score(y_vldt, y_pred))\n# f1_lst.append(f1_score(y_vldt, y_pred))\n# auc_lst.append(roc_auc_score(y_vldt, y_pred))\n# # search result\n# score_dict = pd.DataFrame(data={\n# 'precision':precision_lst, # TP/(TP+FP)\n# 'recall':recall_lst, # TP/(TP+FN)\n# 'f1':f1_lst, # F1 Score\n# 'auc':auc_lst # AUC\n# }, index=grid_point_lst)\n# estimator_dict = pd.DataFrame(data={'estimator':estimator_lst}, index=grid_point_lst)\n# return score_dict, estimator_dict\n\n# def run_grid_search(base_model_id, exp_params_id): \n# with timer_memory('grid_search %'.format(exp_params_id)):\n# lgb_model, search_params = get_model_and_search_params(base_model_id, exp_params_id)\n# score_dict, estimator_dict = grid_search(X_train, y_train, g_X_vldt, g_y_vldt, lgb_model, search_params)\n# return score_dict","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Run Grid Search","metadata":{}},{"cell_type":"code","source":"# g_grid_search_results = {\n# 'gbdt_base_001_exp_002' : run_grid_search('gbdt_base_001', 'gbdt_base_001_exp_002'),\n# 'gbdt_base_001_exp_005' : run_grid_search('gbdt_base_001', 'gbdt_base_001_exp_005'),\n# 'gbdt_base_001_exp_010' : run_grid_search('gbdt_base_001', 'gbdt_base_001_exp_010')\n# }\n\n# def plot_grid_search_score(exp_id, grid_results=g_grid_search_results):\n# grid_results[exp_id].plot.barh()\n# return grid_results[exp_id]","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Grid Search Results","metadata":{}},{"cell_type":"code","source":"# plot_grid_search_score('gbdt_base_001_exp_002')","metadata":{"_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot_grid_search_score('gbdt_base_001_exp_005')","metadata":{"_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot_grid_search_score('gbdt_base_001_exp_010')","metadata":{"_kg_hide-output":false,"trusted":true},"execution_count":null,"outputs":[]}]}