{"cells":[{"metadata":{"_uuid":"a662414b110cfcf1aa08b147c577ca576e9f7710"},"cell_type":"markdown","source":"# Intro\n\nWe will briefly illustrate how to calculate *Weight of Evidence* and *Information Value* in Python.\n\n**Warning**: This is going to be super hacky and not optimal in the slightest. It is purely for illustration. \n\nIf anyone cares to optimize with NumPy/Vectorization and replacing the for-loop with some `.apply()` method, I would be most appreciative. I'll do the same if I get to it! :)\n\n* Quick shout-out to the inspiration for this kernel: https://www.kaggle.com/pranav84/talkingdata-with-breaking-bad-feature-engg\n\nHere we go..."},{"metadata":{"_uuid":"53e74f791ad87431b4ad83415182145095bb5619"},"cell_type":"markdown","source":"# Import Packages"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","collapsed":true,"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"f40f0ec3e5fed0715ed382e949412cabd6d2d888"},"cell_type":"markdown","source":"# Load the Data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Load subset of the training data\nX_train = pd.read_csv('../input/train.csv', skiprows=range(1,1000000), nrows=1000000, parse_dates=['click_time'])\n\n# Show the head of the table\nX_train.head()","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"2eee0578970e2f667a9d612fdf64cc26dcdcdfce"},"cell_type":"markdown","source":"# Brief Feature Engineering: *ip_app_nextClick*\n\nWe'll quickly create a feature for illustrating our Information Value tests. \n\nI'm going to borrow some code from this kernel: https://www.kaggle.com/nanomathias/feature-engineering-importance-testing"},{"metadata":{"trusted":true,"_uuid":"03fe64e749f0e380b7d56e87750666dc10351d58"},"cell_type":"code","source":"GROUP_BY_NEXT_CLICKS = [{'groupby': ['ip', 'app']}]\n\n# Calculate the time to next click for each group\nfor spec in GROUP_BY_NEXT_CLICKS:\n    \n    # Name of new feature\n    new_feature = '{}_nextClick'.format('_'.join(spec['groupby']))    \n    \n    # Unique list of features to select\n    all_features = spec['groupby'] + ['click_time']\n    \n    # Run calculation\n    print(f\">> Grouping by {spec['groupby']}, and saving time to next click in: {new_feature}\")\n    X_train[new_feature] = X_train[all_features].groupby(spec['groupby']).click_time.transform(lambda x: x.diff().shift(-1)).dt.seconds\n    \nX_train.head()","execution_count":16,"outputs":[]},{"metadata":{"_uuid":"fe0c343571f4dfbfa8f78219af5d5550b2459787"},"cell_type":"markdown","source":"# Calculate Information Value\n\nHere is the *super hacky* function that does all of the work:"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"12c4e89a32bf77610f83b49f6ba5dbcbdb73c78c"},"cell_type":"code","source":"# Calculate information value\ndef calc_iv(df, feature, target, pr=False):\n    \"\"\"\n    Set pr=True to enable printing of output.\n    \n    Output: \n      * iv: float,\n      * data: pandas.DataFrame\n    \"\"\"\n\n    lst = []\n\n    df[feature] = df[feature].fillna(\"NULL\")\n\n    for i in range(df[feature].nunique()):\n        val = list(df[feature].unique())[i]\n        lst.append([feature,                                                        # Variable\n                    val,                                                            # Value\n                    df[df[feature] == val].count()[feature],                        # All\n                    df[(df[feature] == val) & (df[target] == 0)].count()[feature],  # Good (think: Fraud == 0)\n                    df[(df[feature] == val) & (df[target] == 1)].count()[feature]]) # Bad (think: Fraud == 1)\n\n    data = pd.DataFrame(lst, columns=['Variable', 'Value', 'All', 'Good', 'Bad'])\n\n    data['Share'] = data['All'] / data['All'].sum()\n    data['Bad Rate'] = data['Bad'] / data['All']\n    data['Distribution Good'] = (data['All'] - data['Bad']) / (data['All'].sum() - data['Bad'].sum())\n    data['Distribution Bad'] = data['Bad'] / data['Bad'].sum()\n    data['WoE'] = np.log(data['Distribution Good'] / data['Distribution Bad'])\n\n    data = data.replace({'WoE': {np.inf: 0, -np.inf: 0}})\n\n    data['IV'] = data['WoE'] * (data['Distribution Good'] - data['Distribution Bad'])\n\n    data = data.sort_values(by=['Variable', 'Value'], ascending=[True, True])\n    data.index = range(len(data.index))\n\n    if pr:\n        print(data)\n        print('IV = ', data['IV'].sum())\n\n\n    iv = data['IV'].sum()\n    # print(iv)\n\n    return iv, data","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"f8cf67445e2755c326f88c77810de69ae546d7e7"},"cell_type":"markdown","source":"This is way too expensive to run on `X_train`. So we'll run it on a sample of the data:"},{"metadata":{"trusted":true,"_uuid":"1baaabf4cb1f5e229822da38cd5ccc39fb12df20"},"cell_type":"code","source":"df = X_train.sample(7777).copy()\n\niv, data = calc_iv(df, 'ip_app_nextClick', 'is_attributed')","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"9b03deb90fe7b881a6d2b9eef2d7e4e726f07136"},"cell_type":"markdown","source":"This falls under tha category of \"too good to be true\" typically:"},{"metadata":{"trusted":true,"_uuid":"6a9dd02091d843fde535dced2f8b7145d662be42"},"cell_type":"code","source":"iv","execution_count":19,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a0def394932809adbfeab2b334adf6be49b0085b"},"cell_type":"code","source":"data.head()","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"3f6069be9f7666e9d9bc0b612f835e2d9f86cc43"},"cell_type":"markdown","source":"Where were the hits?"},{"metadata":{"trusted":true,"_uuid":"8bee8ddc75fee1695077615c596c85bacd252d30"},"cell_type":"code","source":"data[data['Bad Rate'] > 0.0]","execution_count":21,"outputs":[]},{"metadata":{"_uuid":"b6d6f6feee6fa4923417b4c3603366a3d4e36fd0"},"cell_type":"markdown","source":"Thanks! Let me know your thoughts.\n\n*-BA*"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}