{"cells":[{"metadata":{"_uuid":"d6254d6d77d67ddba5ac7b446f0930acee4210c7","_cell_guid":"b80bdc99-81c6-437c-8616-34b4f7f06024"},"cell_type":"markdown","source":"Sometimes you want to get an idea of whether a feature is worth exploring in further depth.  Since running a whole model to get each feature's  impact on a huge dataset can be time consuming, getting some basic visualizations on feature class distribution can help prioritize which features to explore.\n\nBelow is a simple method that allows to run top level screening test on multiple numeric feature ideas at once.  Basically it's an additional layer of EDA...\n\nFor this example I'm going to only use the first 10000000 rows of the train set.  You can obviously do it on your own subsamples or full data depending on your computer power...   The idea is to get a set that is representative of the features you are investigating, and is large enough.  (eg:  if you are figuring out variances by hour, you need a sample that has multiple hours in it to be meaningful).\n\nFor this kernel, features and helper functions  are mostly taken from the public kernels for this competition, such as https://www.kaggle.com/aharless/kaggle-runnable-version-of-baris-kanber-s-lightgbm .  You can check any other set of feature in similar way."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":false},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport datetime\nimport os\nimport gc\n\nimport time\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"272434408c4d6540020a477fd2a6114c7c52e331","_cell_guid":"5e56a132-2d11-4cd6-a828-a60b6702ace8","trusted":false,"collapsed":true},"cell_type":"code","source":"dtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        }\n\ncolumns = ['ip', 'app', 'device', 'os', 'channel', 'is_attributed', 'click_time']\ntrain_smp = pd.read_csv('../input/train.csv', dtype=dtypes, nrows=10000000, parse_dates=['click_time'], usecols=columns  )\ntrain_smp.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bdd2cff72e3c47a290f8ebee1d8f10be1676a4ba"},"cell_type":"markdown","source":"These are some functions I use to come up with various groupings.  Other functions can include means, variances, and any other combo you think of."},{"metadata":{"_uuid":"380640d233acd62ce5d0cf0ef5ea61beb884d413","collapsed":true,"_cell_guid":"1d3940db-344f-4dc2-bcb0-84505024593d","trusted":false},"cell_type":"code","source":"################# HELPER AGGREGATION FUNCTIONS ################\n#total count features\ndef do_count( df, group_cols, agg_name, agg_type='uint16', show_max=False, show_agg=True ):\n    if show_agg:\n        print( \"Aggregating by \", group_cols , '...' )\n    gp = df[group_cols][group_cols].groupby(group_cols).size().rename(agg_name).to_frame().reset_index()\n    df = df.merge(gp, on=group_cols, how='left')\n    del gp\n    if show_max:\n        print( agg_name + \" max value = \", df[agg_name].max() )\n    df[agg_name] = df[agg_name].astype(agg_type)\n    gc.collect()\n    return( df )\n\n#unique count features\ndef do_countuniq( df, group_cols, counted, agg_name, agg_type='uint16', show_max=False, show_agg=True ):\n    if show_agg:\n        print( \"Counting unqiue \", counted, \" by \", group_cols , '...' )\n    gp = df[group_cols+[counted]].groupby(group_cols)[counted].nunique().reset_index().rename(columns={counted:agg_name})\n    df = df.merge(gp, on=group_cols, how='left')\n    del gp\n    if show_max:\n        print( agg_name + \" max value = \", df[agg_name].max() )\n    df[agg_name] = df[agg_name].astype(agg_type)\n    gc.collect()\n    return( df )\n    \n#cummulative count features    \ndef do_cumcount( df, group_cols, counted, agg_name, agg_type='uint16', show_max=False, show_agg=True ):\n    if show_agg:\n        print( \"Cumulative count by \", group_cols , '...' )\n    gp = df[group_cols+[counted]].groupby(group_cols)[counted].cumcount()\n    df[agg_name]=gp.values\n    del gp\n    if show_max:\n        print( agg_name + \" max value = \", df[agg_name].max() )\n    df[agg_name] = df[agg_name].astype(agg_type)\n    gc.collect()\n    return( df )\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f700ce3a37979c7dd1469f64b6d02dad48359457"},"cell_type":"markdown","source":"Generate a set of features want to get a feel for:"},{"metadata":{"_uuid":"5ccb05270a61aa1389fbbf17b86e4e7c82ffb4a1","_cell_guid":"caf632b8-4a31-4bef-a535-60c5502b61c1","trusted":false,"collapsed":true},"cell_type":"code","source":"######## Data Preprocessing  ####\ndef prep_data(df):\n    \n    print('Extracting new features...')\n    \n    #time prep\n    df['hour'] = pd.to_datetime(df.click_time).dt.hour.astype('uint8')\n    df['day'] = pd.to_datetime(df.click_time).dt.day.astype('uint8')\n    df['min'] = pd.to_datetime(df.click_time).dt.minute.astype('uint8')\n\n    ## UNIQUE COUNTS FEATURES:\n    df = do_countuniq( df, ['ip'], 'app', 'uq_app_per_ip', show_max=True ); gc.collect()\n    df = do_countuniq( df, ['ip', 'app'], 'os', 'uq_os_per_ip_app', show_max=True ); gc.collect()\n    df = do_countuniq( df, ['ip'], 'device', 'uq_device_per_ip', show_max=True ); gc.collect()\n    df = do_countuniq( df, ['app'], 'channel', 'uq_channels_per_app', show_max=True ); gc.collect()\n    df = do_countuniq( df, ['ip'], 'channel', 'uq_channel_per_ip', show_max=True ); gc.collect()\n\n    ## TOTALS COUNTS FEATURES:\n    df = do_count( df, ['ip', 'day'], 'ip_per_day_count', show_max=True ); gc.collect()\n    df = do_count( df, ['ip', 'app'], 'ip_app_count', show_max=True ); gc.collect()\n    df = do_count( df, ['ip', 'app', 'os'], 'ip_app_os_count', show_max=True ); gc.collect()\n    df = do_count( df, ['ip', 'channel', 'hour'], 'ip_channel_per_hour', show_max=True ); gc.collect()\n\n    ##### CUMMULATIVE COUNTS FEATURES:\n    df = do_cumcount( df, ['ip', 'device', 'os'], 'app', 'cumcount_ip_dev_os', show_max=True ); gc.collect()\n\n    print('*'*30)\n    print('after data prep:')\n    print(df.head())\n    print('finished feature generation')\n    \n    return (df)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"_uuid":"c263487ac0d50ab5e3fcab8ef864d7f2a422411d","_cell_guid":"22ae8547-a1b8-4eb8-b049-d2549005b5fa","trusted":false,"collapsed":true},"cell_type":"code","source":"#create new features\ntrain_smp = prep_data(train_smp)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9fea76f3fe2648325ccde9f213230357c094a7ba","_cell_guid":"376402f0-311f-4b1d-8552-2793bcdd9a1b"},"cell_type":"markdown","source":"Now can visualize our features by class.   Total counts tend to have high range of values in this set, so I group them separately to plot them with log of counts, otherwise the patterns get too squished on the lower end of the spectrum."},{"metadata":{"_uuid":"134914eda2270343ccc0d3add9ec270dec08e380","_cell_guid":"ec750dd5-3858-4846-bbc0-11d7d1d6628a","trusted":false,"collapsed":true},"cell_type":"code","source":"unique_count_features = ['uq_app_per_ip', 'uq_os_per_ip_app', 'uq_device_per_ip', 'uq_channels_per_app', 'uq_channel_per_ip']\n\nother_count_features = ['ip_per_day_count', 'ip_app_count', 'ip_app_os_count', 'ip_channel_per_hour', 'cumcount_ip_dev_os']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"157f6f677852583588ffefde879fdbade121eb2b","_cell_guid":"326a24f0-2aa5-4aa9-9697-b8a66bda8bd7"},"cell_type":"markdown","source":"Let's see how these  features distributed in each of the classes.  For each I'm going to look at a boxplot (for basic distribution grasp) and a violinplot (to see where the data bulks up).  "},{"metadata":{"scrolled":false,"_uuid":"96c3982152837241feb7298dee1c97f2c398de5b","_cell_guid":"2c8a5279-3f6e-4c36-8845-6dba49c3c6a8","trusted":false,"collapsed":true},"cell_type":"code","source":"#make wider graphs\nsns.set(rc={'figure.figsize':(12,5)});\nplt.figure(figsize=(12,5));\n\nfor fea in unique_count_features:\n    sns.boxplot(train_smp.is_attributed, train_smp[fea])\n    title_2 = 'PLOT OF: ' +fea\n    plt.title(title_2)\n    plt.show()\n    sns.violinplot(train_smp.is_attributed, train_smp[fea])\n    title_2 = 'PLOT OF:  ' +fea\n    plt.title(title_2)\n    plt.show()\n    gc.collect()\n    print('*'*70)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8925c675d02560972b410e174af908afb01206c3","_cell_guid":"13eeb0d5-e89d-4d3e-8018-df93f70e4d21"},"cell_type":"markdown","source":"To visualize total count features that have large range of values, it make sense to look at log plots (taking log of counts), to zoom in on the difference, otherwise the data looks very squished at the lower end of the spectrum."},{"metadata":{"_uuid":"5f1422fe64e1b683299d80b5760b64fe8218bd36","_cell_guid":"eb30ff2e-0883-4e75-bd5c-4cb62cb4e1c9","trusted":false,"collapsed":true},"cell_type":"code","source":"#visualize distribution by remaining count features, normal boxplot, log boxplot and log violinplot\nfor fea in other_count_features:\n    sns.boxplot(train_smp.is_attributed, train_smp[fea])\n    title = 'REGULAR BOXPLOT PLOT OF: ' +fea\n    plt.title(title)\n    plt.show()\n    sns.boxplot(train_smp.is_attributed, np.log(train_smp[fea]+1))\n    title = 'LOG BOXPLOT PLOT OF: ' +fea\n    plt.title(title)\n    plt.show()\n    sns.violinplot(train_smp.is_attributed, np.log(train_smp[fea]+1))\n    title = 'LOG VIOLINPLOT PLOT OF:  ' +fea\n    plt.title(title_2)\n    plt.show()\n    gc.collect()\n    print('*'*70)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"612ccbebcd94f91ad8b55562ab0ae8cd099a5348","_cell_guid":"38f3dac3-bc85-441d-9f70-366c91868fa8"},"cell_type":"markdown","source":"Most of the features above tend to have different distribution, and are good candidates for experimenting, though 'uq_device_per_ip' doesn't look as separated to me, so I would  put it lower on testing priority. \n\nNow let's say you wanted to use minutes as a predictor based on given dataset:"},{"metadata":{"_uuid":"cc8ab20e926523dcccdd4279b1534edf55d3a2fe","_cell_guid":"04d157c2-e260-4609-8f80-e3decb37c11a","trusted":false,"collapsed":true},"cell_type":"code","source":"#visualize distribution of attributions by minute\nsns.boxplot(train_smp.is_attributed, train_smp['min'])\nplt.title('Boxplot of Minute distribution')\nplt.show()\nsns.violinplot(train_smp.is_attributed, train_smp['min'])\nplt.title('Violinplot of Minute distribution')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc51d118d27c3391a765faa1daf8e8a205c6e85d","_cell_guid":"b29c5a34-9168-45e4-a5e6-362d3dcd0ff6"},"cell_type":"markdown","source":"There really isn't any difference in distribution of attributions by minute.  So doesn't really make sense to dig into that one by itself for consistent results.\n\nOn the other hand a popular feature in the kernels and forums is time deltas.  I'm going to use the method from https://www.kaggle.com/asydorchuk/nextclick-calculation-without-hashing-trick kernel to generate it, and fill in the missing values with some large value out of range."},{"metadata":{"_uuid":"694f3db124891d487a8b7a8edeaa0bf8c4e7122f","_cell_guid":"c4121431-d24f-4401-bdf7-71e041c469ce","trusted":false,"collapsed":true},"cell_type":"code","source":"#generate next_click feature\ntrain_smp['click_time'] = (train_smp['click_time'].astype(np.int64) // 10 ** 9).astype(np.int32)\ntrain_smp['next_click'] = (train_smp.groupby(['ip', 'app', 'device', 'os']).click_time.shift(-1) - train_smp.click_time).astype(np.float32)\ntrain_smp['next_click'] = train_smp['next_click'].fillna(360000000).astype('uint32')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4fadca48692e0688bdb0c14c5c4f3f5a129ba67","_cell_guid":"96f32676-d75c-45f1-9809-a04c20ba0176","trusted":false,"collapsed":true},"cell_type":"code","source":"#visualize next_click\nsns.boxplot(train_smp.is_attributed, np.log(train_smp['next_click']+1))\nplt.title('LOG Boxplot of Next_Click')\nplt.show()\nsns.violinplot(train_smp.is_attributed, np.log(train_smp['next_click']+1))\nplt.title('LOG Violinplot of Next_click')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67dd4dfc375bf4b4ddc71b1aac4fa17ccd0678b5","_cell_guid":"858b8f24-30fc-4182-a031-dd74f03eff3b"},"cell_type":"markdown","source":"As you can see this 'next_click' feature has significantly different distribution for the two classes, and this is only based on first 10M rows!  Hense its popularity in the models...\n\n\n**Disclaimers**:  \n- it is really important that the analysis is run on representative dataset.  As mentioned before,  if you only pulling rows from one hour of data and trying to get idea of impact of hourly variances, you'll get meaningless results.   \n- this method doesn't guarantee that the feature will improve your model, as it may be affected by various interactions with other features.  But I think it's a good 'is-it-even-worth-it' method for when you are strapped for time and resources.  Plus it allows to screen multiple feature ideas in bulk.\n\nHappy Feature Engineering!"},{"metadata":{"_uuid":"b4c7a68c0d3b93553a4be8780288724e0690874a","collapsed":true,"_cell_guid":"0800d438-4f41-49c9-b7a0-b5a8ba970401","trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}