{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport itertools\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ndef read_data(cols):\n    \n    print('Reading data...')\n    \n    df = pd.read_parquet('../input/amex-data-integer-dtypes-parquet-format/train.parquet', columns=cols)\n    \n    # simplify cus_id\n    unique_cus_ids = df.customer_ID.unique()\n    assignment     = dict(zip(unique_cus_ids, list(range(len(unique_cus_ids)))))\n    df.customer_ID = df.customer_ID.apply(lambda x: assignment[x]).astype('int32')\n    \n    print('shape of data:', df.shape)\n    \n    return df\n\n# method for Information Value\ndef iv_woe(data, target, bins=20, show_woe=False):\n    \n    #Empty Dataframe\n    newDF,woeDF = pd.DataFrame(), pd.DataFrame()\n    \n    #Extract Column Names\n    cols = data.columns\n    \n    #Run WOE and IV on all the independent variables\n    for ivars in cols[~cols.isin([target])]:\n\n        if (data[ivars].dtype.kind in 'bifc') and (len(np.unique(data[ivars]))>20):\n            binned_x = pd.qcut(data[ivars], bins,  duplicates='drop')\n            d0 = pd.DataFrame({'x': binned_x, 'y': data[target]})\n        else:\n            d0 = pd.DataFrame({'x': data[ivars], 'y': data[target]})\n            \n        d0 = d0.astype({\"x\": str})\n        d = d0.groupby(\"x\", as_index=False, dropna=False).agg({\"y\": [\"count\", \"sum\"]})\n        d.columns = ['Cutoff', 'N', 'Events']\n        d['% of Events'] = np.maximum(d['Events'], 0.5) / d['Events'].sum()\n        d['Non-Events'] = d['N'] - d['Events']\n        d['% of Non-Events'] = np.maximum(d['Non-Events'], 0.5) / d['Non-Events'].sum()\n        d['WoE'] = np.log(d['% of Non-Events']/d['% of Events'])\n        d['IV'] = d['WoE'] * (d['% of Non-Events']-d['% of Events'])\n        d.insert(loc=0, column='Variable', value=ivars)\n        print(\"Information value of \" + ivars + \" is \" + str(round(d['IV'].sum(),6)))\n        temp =pd.DataFrame({\"Variable\" : [ivars], \"IV\" : [d['IV'].sum()]}, columns = [\"Variable\", \"IV\"])\n        newDF=pd.concat([newDF,temp], axis=0)\n        woeDF=pd.concat([woeDF,d], axis=0)\n\n        #Show WOE Table\n        if show_woe == True:\n            print(d)\n    return newDF, woeDF","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-15T14:34:30.817104Z","iopub.execute_input":"2022-08-15T14:34:30.819631Z","iopub.status.idle":"2022-08-15T14:34:31.474915Z","shell.execute_reply.started":"2022-08-15T14:34:30.819587Z","shell.execute_reply":"2022-08-15T14:34:31.473723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# How to use WoE for categorial variables","metadata":{}},{"cell_type":"markdown","source":"I haven been talking in my previous [notebook](https://www.kaggle.com/code/gzguevara/smart-brute-force-feature-engineering) about \"Information Values\" for \"smart brute force feature\" engineering. Check out \"brute force engineering\" [here](https://www.kaggle.com/code/thedevastator/amex-bruteforce-feature-engineering).  In another [notebook](https://www.kaggle.com/code/gzguevara/extract-all-information-from-categorial-features), I proposed to extract all provided information given by categorial features foucusing on aggregation over time.\n\nToday I would like to combine those concepts! \n\nWhen working with tree bosting algorithms and categorial variables people often appleal to CatBoost or to LGMB. Both libraies provide well-developed categorial variable support. If you check the LGBM [documentation](https://lightgbm.readthedocs.io/en/v3.3.2/Features.html#optimal-split-for-categorical-features) on that topic you will find the following explaination:\n\n*The basic idea is to sort the categories according to the training objective at each split. More specifically, LightGBM sorts the histogram (for a categorical feature) according to its accumulated values (sum_gradient / sum_hessian) and then finds the best split on the sorted histogram.*\n\nThe same concept can be mimicked by a Weights of Evidence (WoE) transformations. A WoE transformation assigns to each class the natural logarithm of the division of the percentage of non-events by the percentage of events. Where \"non-events\" stands for customers which did not default, while \"events\" are customer which did default on their payments.\n\n![image.png](attachment:d6a11ee6-c36e-4352-a0fb-927f9bce7a25.png)\n\nIn this notebook we will reproduce the same logic of LGBM and CatBoost, using Weights of Evidence. Let's get started!","metadata":{},"attachments":{"d6a11ee6-c36e-4352-a0fb-927f9bce7a25.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAOUAAABFCAIAAAA6t7avAAAMbmlDQ1BJQ0MgUHJvZmlsZQAASImVVwdYU8kWnluSkJCEEkBASuhNEKkBpITQAkgvgo2QBBJKjAlBxV4WFVy7iIANXRVRbCsgduzKotj7YkFFWRd1saHyJiSg677yvfN9c++fM2f+U+5M7j0AaH3gSaV5qDYA+ZICWUJ4MHN0WjqT9BSggA50gBOw5vHlUnZcXDSAMnD/u7y7ARDl/aqzkuuf8/9VdAVCOR8AZCzEmQI5Px/i4wDgVXyprAAAolJvNblAqsSzIdaTwQAhXqXE2Sq8XYkzVfhwv01SAgfiywBoUHk8WTYA9HtQzyzkZ0Me+meIXSUCsQQArWEQB/BFPAHEytiH5edPVOJyiO2hvRRiGA9gZX7Hmf03/sxBfh4vexCr8uoXjRCxXJrHm/p/luZ/S36eYsCHLRxUkSwiQZk/rOGt3IlRSkyFuEuSGROrrDXEH8QCVd0BQCkiRUSyyh414cs5sH7AAGJXAS8kCmITiMMkeTHRan1mljiMCzHcLegUcQE3CWJDiBcK5aGJapuNsokJal9ofZaMw1brz/Fk/X6Vvh4ocpPZav43IiFXzY/Ri0RJqRBTILYuFKfEQEyH2EWemxilthlZJOLEDNjIFAnK+K0hThBKwoNV/FhhliwsQW1fki8fyBfbKBJzY9R4X4EoKUJVH+wUn9cfP8wFuyyUsJMHeITy0dEDuQiEIaGq3LHnQklyoprng7QgOEG1FqdI8+LU9rilMC9cqbeE2ENemKhei6cUwM2p4sezpAVxSao48aIcXmScKh58GYgGHBACmEABRyaYCHKAuLWroQv+Us2EAR6QgWwgBM5qzcCK1P4ZCbwmgiLwB0RCIB9cF9w/KwSFUP9lUKu6OoOs/tnC/hW54CnE+SAK5MHfiv5VkkFvKeAJ1Ij/4Z0HBx/GmweHcv7f6we03zRsqIlWaxQDHplaA5bEUGIIMYIYRnTAjfEA3A+PhtcgONxwFu4zkMc3e8JTQhvhEeE6oZ1we4J4ruyHKEeBdsgfpq5F5ve1wG0hpycejPtDdsiMG+DGwBn3gH7YeCD07Am1HHXcyqowf+D+WwbfPQ21HdmVjJKHkIPI9j+upDvSPQdZlLX+vj6qWDMH680ZnPnRP+e76gvgPepHS2whth87i53AzmOHsQbAxI5hjVgLdkSJB3fXk/7dNeAtoT+eXMgj/oc/ntqnspJy11rXTtfPqrkC4ZQC5cHjTJROlYmzRQVMNnw7CJlcCd9lGNPN1c0NAOW7RvX39Ta+/x2CGLR80837HQD/Y319fYe+6SKPAbDXGx7/g9909iwAdDQBOHeQr5AVqnS48kKA/xJa8KQZATNgBexhPm7AC/iBIBAKIkEsSAJpYDyssgjucxmYDKaDOaAYlIJlYDWoABvAZrAd7AL7QAM4DE6AM+AiuAyug7tw93SAl6AbvAO9CIKQEBrCQIwQc8QGcULcEBYSgIQi0UgCkoZkINmIBFEg05F5SCmyAqlANiE1yF7kIHICOY+0IbeRh0gn8gb5hGIoFdVDTVFbdDjKQtloFJqEjkOz0UloETofXYKWo9XoTrQePYFeRK+j7ehLtAcDmCZmgFlgzhgL42CxWDqWhcmwmVgJVoZVY3VYE3zOV7F2rAv7iBNxBs7EneEOjsCTcT4+CZ+JL8Yr8O14PX4Kv4o/xLvxrwQawYTgRPAlcAmjCdmEyYRiQhlhK+EA4TQ8Sx2Ed0Qi0YBoR/SGZzGNmEOcRlxMXEfcTTxObCM+JvaQSCQjkhPJnxRL4pEKSMWktaSdpGOkK6QO0gcNTQ1zDTeNMI10DYnGXI0yjR0aRzWuaDzT6CVrk23IvuRYsoA8lbyUvIXcRL5E7iD3UnQodhR/ShIlhzKHUk6po5ym3KO81dTUtNT00YzXFGvO1izX3KN5TvOh5keqLtWRyqGOpSqoS6jbqMept6lvaTSaLS2Ilk4roC2h1dBO0h7QPtAZdBc6ly6gz6JX0uvpV+ivtMhaNlpsrfFaRVplWvu1Lml1aZO1bbU52jztmdqV2ge1b2r36DB0RujE6uTrLNbZoXNe57kuSddWN1RXoDtfd7PuSd3HDIxhxeAw+Ix5jC2M04wOPaKenR5XL0evVG+XXqtet76uvod+iv4U/Ur9I/rtBpiBrQHXIM9gqcE+gxsGn4aYDmEPEQ5ZNKRuyJUh7w2HGgYZCg1LDHcbXjf8ZMQ0CjXKNVpu1GB03xg3djSON55svN74tHHXUL2hfkP5Q0uG7ht6xwQ1cTRJMJlmstmkxaTH1Mw03FRqutb0pGmXmYFZkFmO2Sqzo2ad5gzzAHOx+SrzY+YvmPpMNjOPWc48xey2MLGIsFBYbLJotei1tLNMtpxrudvyvhXFimWVZbXKqtmq29rcepT1dOta6zs2ZBuWjchmjc1Zm/e2draptgtsG2yf2xnace2K7Grt7tnT7APtJ9lX219zIDqwHHId1jlcdkQdPR1FjpWOl5xQJy8nsdM6p7ZhhGE+wyTDqofddKY6s50LnWudH7oYuES7zHVpcHk13Hp4+vDlw88O/+rq6ZrnusX17gjdEZEj5o5oGvHGzdGN71bpds2d5h7mPsu90f21h5OH0GO9xy1PhucozwWezZ5fvLy9ZF51Xp3e1t4Z3lXeN1l6rDjWYtY5H4JPsM8sn8M+H329fAt89/n+6efsl+u3w+/5SLuRwpFbRj72t/Tn+W/ybw9gBmQEbAxoD7QI5AVWBz4KsgoSBG0NesZ2YOewd7JfBbsGy4IPBL/n+HJmcI6HYCHhISUhraG6ocmhFaEPwizDssNqw7rDPcOnhR+PIERERSyPuMk15fK5NdzuSO/IGZGnoqhRiVEVUY+iHaNl0U2j0FGRo1aOuhdjEyOJaYgFsdzYlbH34+ziJsUdiifGx8VXxj9NGJEwPeFsIiNxQuKOxHdJwUlLk+4m2ycrkptTtFLGptSkvE8NSV2R2j56+OgZoy+mGaeJ0xrTSekp6VvTe8aEjlk9pmOs59jisTfG2Y2bMu78eOPxeeOPTNCawJuwP4OQkZqxI+MzL5ZXzevJ5GZWZXbzOfw1/JeCIMEqQafQX7hC+CzLP2tF1vNs/+yV2Z2iQFGZqEvMEVeIX+dE5GzIeZ8bm7stty8vNW93vkZ+Rv5Bia4kV3JqotnEKRPbpE7SYmn7JN9Jqyd1y6JkW+WIfJy8sUAPftS3KOwVPykeFgYUVhZ+mJwyef8UnSmSKS1THacumvqsKKzol2n4NP605ukW0+dMfziDPWPTTGRm5szmWVaz5s/qmB0+e/scypzcOb/NdZ27Yu5f81LnNc03nT97/uOfwn+qLaYXy4pvLvBbsGEhvlC8sHWR+6K1i76WCEoulLqWlpV+XsxffOHnET+X/9y3JGtJ61KvpeuXEZdJlt1YHrh8+wqdFUUrHq8ctbJ+FXNVyaq/Vk9Yfb7Mo2zDGsoaxZr28ujyxrXWa5et/VwhqrheGVy5u8qkalHV+3WCdVfWB62v22C6oXTDp43ijbc2hW+qr7atLttM3Fy4+emWlC1nf2H9UrPVeGvp1i/bJNvatydsP1XjXVOzw2TH0lq0VlHbuXPszsu7QnY11jnXbdptsLt0D9ij2PNib8beG/ui9jXvZ+2v+9Xm16oDjAMl9Uj91PruBlFDe2NaY9vByIPNTX5NBw65HNp22OJw5RH9I0uPUo7OP9p3rOhYz3Hp8a4T2SceN09ovnty9Mlrp+JPtZ6OOn3uTNiZk2fZZ4+d8z93+Lzv+YMXWBcaLnpdrG/xbDnwm+dvB1q9WusveV9qvOxzualtZNvRK4FXTlwNuXrmGvfaxesx19tuJN+4dXPszfZbglvPb+fdfn2n8E7v3dn3CPdK7mvfL3tg8qD6d4ffd7d7tR95GPKw5VHio7uP+Y9fPpE/+dwx/yntadkz82c1z92eH+4M67z8YsyLjpfSl71dxX/o/FH1yv7Vr38G/dnSPbq747Xsdd+bxW+N3m77y+Ov5p64ngfv8t/1vi/5YPRh+0fWx7OfUj896538mfS5/IvDl6avUV/v9eX39Ul5Ml7/pwAGB5qVBcCbbQDQ0gBgwL6NMkbVC/YLoupf+xH4T1jVL/aLFwB18Ps9vgt+3dwEYM8W2H5Bfi3Yq8bRAEjyAai7++BQizzL3U3FRYV9CuFBX99b2LORVgLwZVlfX291X9+XzTBY2Dsel6h6UKUQYc+wMfRLZn4m+Dei6k+/y/HHO1BG4AF+vP8LwzKQ1fn5IUQAAAA4ZVhJZk1NACoAAAAIAAGHaQAEAAAAAQAAABoAAAAAAAKgAgAEAAAAAQAAAOWgAwAEAAAAAQAAAEUAAAAAzfj6ZAAAC3dJREFUeAHtXT9sE0kX33w6CTqgI4KLKGJBigiQIoJJgVJBopOcAhGgANEkQhRffAiRJlIQBZHQ4TQIkgb0FYigE8TSyYYKKIzJKQVSioAcCTAXhZIOOn+/mdk/s+vZnVn/ydny2yKeffPmzdvf/vbN29mdbFelUrFoIwTaBIH/tImf5CYhwBAgvhIP2gkB4ms7nS3ylfhKHGgnBIiv7XS2yFfiK3GgnRD4pZ2cbVVfl5eXs9ns58+fDxw48PDhw1Z1s1X8unz5MrDau3dvKpUaGxvbuXOnuWddNP9qDla15ocPH4C+gB5kFVu1GklkBEBWbN++fcNFDgDv379//PhxWSGqDL7SVhsCt2/fPnLkSLFYrK05tQIC6+vrIOv09LQhGpS/Rl3MEXWzs7NfvnwBWWPEhghznVp16NAhYLhjx44rV66YYEB8NUEpqPP+/fuXL19mMplYuVfQCu07CODiR4bw4sULRxD6S/lrKDRhFT9//hweHkbWhWQgTIfkcRFAOgtUEWt3794d0ZbiawQ46qonT55gFCOyqtGpVYp71tOnTz969CjaAPE1Gh9FLTKBU6dOKSpIVB8Cg4ODKysr0TaIr9H4KGoxBYP4qqggUX0IYMjCjUG0Dcpfo/FR1O7Zs+fTp0/RaZaiGYkMEOjq0hBSU23QRcepaDHtOEQad8BabCkfaBzYZKn5CBBfm48x9dA4BIivjcOSLDUfgU7g6+q9C9jurTYfTeqh2Qh0Al+bjSHZ3z4EdO+/IjbdLfSM/zGX6pad4mJr6PfHVwdk8VZ2+tpSOSgWyp5emDlPQ5SCdoL1Lbe/MX8ikS7CrYlcZWHEdi8/2XWrr/R2qtfQXcdIMhOjkaHt9lfT8XVgcMgqFIp/b6Vkwq6uFNihF1ZWrw7IhN38p2xZQ4OeSBDYR2wmunZhqZqM1TRuM3jzd9L9ucrbETDu4vzGCGdofnLUylWMyWpZMFKU6d5mGMjusitvfca7cuW6WsvafGDf/h7LKv+zKXewtflV7H7d3JLlnMY9+/c5stV7iLagoS8Kd6fm/hjvsQp3p7O+xk6jtv3d+LgmfO892F9cL6GMEwa2upHW9MiSfQlT1Y7T0/K1+1gShEUk9aDZ+rtYtobGQboyAq8kZzTuSR6zM4et7J+IwkNn5MAslLtTZ4ZwEfgae2aaX0KId27AxL0Yux+7cKHOCwg0tdY+boCmH9cY5TbmLyLehrIVZMbsuNgm8/ygmWh00SqmExDbMoEGq5nMI7UQ24l5dGNvCjv8SgnTd9qFN2fdeL0z+05/5n0xTZYbLY7CY9FcauzYC/hisKvlq1VFWE7Xnv3H9v3q55yQe3RlrPYlB5I7LM2ojbAywQTN7L+x2fb1T7D2Lk9shGflpWuxjUjHZI1cz1iMaol0/8yUFclWnDto5cRL9aXM2ignSO/U20puwkLmiooqoi+OIg9mLXITxfQdl+EKO8Iplb7sLivnJ/3NOZFGxiasxWXRgbXx19Ni8uxvLPtWKguLVX2xQyllkiyTr1RY8i6uXuY+NvNsXpj3/ur5ahPWG/pZksqiqOCcmykE6GrxXFZKDrw+WUmVZvg1QvYGrj5Wb4E7wpDmkrhcZtfT77Y1lqPgElp6Jg0kkrJRkZ0kvi0kBFsTiDJsC8YTRoIJN/T2Ts1MiMgc2ctEzj7NjE88kHMyhdpR6Afsb8zfWkxmrtt3hsyN4tO/ELklwnp0DVPmNvV9MTX3Igj4EWPXgK+CsO7ozZPUX/dh0OeEdTMFzk8uj9G9rIrwJgfM5s+X+jJrkaNYlnddyr7FKyOWPD1bskkLAuf607gBk2yU1tksgrcl+pIi4/VERqWYdqQR2R3vRfLBryqWi4jNJSy/sGbc20WlspGnFq5lPo4orl4zA0JLNz/AtbrZ0G9PEvB7LXsKgAdJe5KA01ieGjDwwh99t3t+wMu0hasi5Bu4rVHBsAm2vsUQaFni1gl09LdhgnVJBN7VdJMV0w6L/1NSt6yonotAapNMLOev92EcmHFm5sKUAxZDd+3ukR4nJg9W5zuh7eQKI77ySFoo8KGf32udETNWLLVdKrOINLDJZrh8dPVGfN/Mrd25mPliUTrmFpzMdZtvN93djv0FTGCtZUoLLN/DxqNmL+jYP2ZLuLj3t7PJ9OjkmDhpfKQ9W5IVuJb+T512ePNb89fF1JuvO1F1MenRNULZ19K/w/KWEf+RBS8yfwPN3i+aertaTMNiunWQJ6/OjBUn7BLu849ZmBrw0ZVnETaZq1kZnPky84Jpsfz1qrn6Nmv62MoSwq7RLgyyuINa8HmCUJNbF3WQ1/5ooE47aF6yTiS60o5zniecnpgK9qJrhLLT3P8rTMA6s3rwjpduIKZ7MdvfRrdnyFc7wq7cswpWz7gzY2UJUi4Vnz2zMMNlR12nS0Fm3MCk/A/BLCt8pstp266/Iwu+M4Hdip+o3oGF1AUs2PrgivRvpX06SjsR+p4HvMQ0gzlCaIVSObwvn3oEFAGXonZN7rd4e5GrFjDq++6peGpbLhQwZeA9J7A77E79lz8Y8E9sYkAXTxGCLI5yk+oIAY6AaXy1h3c2AyQ9boUJkSnIzwkkZPEs63GKZZx4AOuJqx/F8jo2PyBpwWTVawuejRYs4c63Bb1qWZcwb1KDb8Z8tTj3FD1oE0qtQqsnpYqDVolqOwEqSyQLRYDWb4VCE1aBOErUDAOnTrkWW+P8tU5HqDkh0AgEiK+NQJFsbBcCxNftQpr6aQQCxNdGoEg2tgsB4ut2IU39NAIB4msjUIxtw31Tyn1PCibwIkjwxcNIw46RWI0iLbZBJaZmaIuFAE5qLH2FMl7K5i8y441mPFsXCo5Moa4WoYGwoq5uitR9Bbsp1vkThGjLFF//hZhCK71qBz2azlRbjUBj4iuPq3Z81UQtVu1sPCxX+FITR1QVZBX6/ljMFJy4rlAW5idyrBHfhK6saTeXRI69arziSNBdtLqmOrpxZ9ZqMTWAxT3R4B8r2zRUtOSaTrVvB3RSkUSmJtMXOmopWy3m9u0p25eDbT2o4zijcVxxLHqRFlviqx7EgIYW04B+9C5jiU1aGK5ioF3t2gB5HJJKRbeac8hRYFJXyWOdx0uvJAy4yv5eXbETeO3umJbLdltW548WW8pfGUv+ta0pK72Uq6xad0lWLPCJr7HgaqwyX+n1P7GaL2Kll9ypyUovd8AWwc5ZVYslWVigylcQjnmv96uV5S6jyuyNbGxYUJmQZ+aimtRXR3ytD786WvO1MzZb7ZVeFlvpdbBXMsrWlCyOOlwQK734PwOQdOQi17/lW4zrVIuqi1hB6NA1QtlppPgVS8l9FVULKn21Dd2pM+HowOb4RtyPHz/qPXApKWSmWHrJNjn3dLpw6gLVAQuOdmDyQLbIE85AxilkvG/Pfmj+at+JCT8lvwJGPV9iluBCdAt6ldM5Vca/fX19z58/p0/EGANmqohPxh09enRrS/oXV1VNKR+ogkQnwEe48fFInRbVx0bg3bt32o/xEl9jw4qPxeGTcbGbUQMdAvj41uHDh6O1iK/R+Chqz5079/r1awxeijoS1YoAvsqbzWaBbbQB4ms0PopafOn0xo0b6bT7TyYUOiSKi8DNmzfHx8e1dwXE17jAMn0RBrQf563FdEe2wXiFbXp6Wnv0ND+ghUit8P379/PnzyPWZjIZ+janGiMDKdIARFaQNZ/Pm8BI8dUAVJUKwAXEJ0+eTCaTs7OzQBwMVimSTIEAaArE5ubmgN6uXbuKxaIJWWGI4qsCzVgi3Hg9ePDgzZs3uL3FF6ZfvXoVq3kHKg8PD4upK8xeXbp0SZuzyhARX2U0qNzqCFA+0OpniPyTESC+ymhQudURIL62+hki/2QEiK8yGlRudQSIr61+hsg/GQHiq4wGlVsdgf8Dwy6dn/qq8+8AAAAASUVORK5CYII="}}},{"cell_type":"code","source":"cat_features = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:34:23.076802Z","iopub.execute_input":"2022-08-15T14:34:23.077291Z","iopub.status.idle":"2022-08-15T14:34:23.109054Z","shell.execute_reply.started":"2022-08-15T14:34:23.077199Z","shell.execute_reply":"2022-08-15T14:34:23.107773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read only categorial features\ndata  = read_data(['customer_ID'] + cat_features)\n# read targets\ntarget = pd.read_csv('../input/amex-default-prediction/train_labels.csv', usecols=['target'])","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:35:58.116388Z","iopub.execute_input":"2022-08-15T14:35:58.116832Z","iopub.status.idle":"2022-08-15T14:36:03.831777Z","shell.execute_reply.started":"2022-08-15T14:35:58.116795Z","shell.execute_reply":"2022-08-15T14:36:03.830469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For WoE we need one point in time, which corresponds to eihter default or non-default. For this notebook let's use the last categorial state and the first categorial state for each customer. \n\nNote: Customers with one record will have the same value for last and first.","metadata":{}},{"cell_type":"code","source":"data = data.groupby('customer_ID').agg(['last', 'first'])\ndata.columns = ['_'.join(x) for x in data.columns]\ndata['target'] = target.target","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-08-15T14:36:04.931882Z","iopub.execute_input":"2022-08-15T14:36:04.932336Z","iopub.status.idle":"2022-08-15T14:36:06.915456Z","shell.execute_reply.started":"2022-08-15T14:36:04.932299Z","shell.execute_reply":"2022-08-15T14:36:06.913924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we apply the WoE transformation! You can read more about it [here](https://www.listendata.com/2015/03/weight-of-evidence-woe-and-information.html#id-b8d17e).","metadata":{}},{"cell_type":"code","source":"info, woe_map = iv_woe(data, 'target')","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:37:16.345538Z","iopub.execute_input":"2022-08-15T14:37:16.345964Z","iopub.status.idle":"2022-08-15T14:37:23.981486Z","shell.execute_reply.started":"2022-08-15T14:37:16.345931Z","shell.execute_reply":"2022-08-15T14:37:23.979958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 'woe_map' tells us what value to assgin to which categorial state.","metadata":{}},{"cell_type":"code","source":"woe_map","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:37:33.666401Z","iopub.execute_input":"2022-08-15T14:37:33.666997Z","iopub.status.idle":"2022-08-15T14:37:33.695652Z","shell.execute_reply.started":"2022-08-15T14:37:33.666942Z","shell.execute_reply":"2022-08-15T14:37:33.694444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's assign those values to our categorial variables!","metadata":{}},{"cell_type":"code","source":"for index, row in woe_map.iterrows():\n    \n    data.loc[data[row.Variable] == int(row.Cutoff), row.Variable] = row.WoE","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:37:38.075351Z","iopub.execute_input":"2022-08-15T14:37:38.076932Z","iopub.status.idle":"2022-08-15T14:37:38.737006Z","shell.execute_reply.started":"2022-08-15T14:37:38.076865Z","shell.execute_reply":"2022-08-15T14:37:38.735781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And BOM! we are done! ","metadata":{}},{"cell_type":"code","source":"data","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:37:40.528839Z","iopub.execute_input":"2022-08-15T14:37:40.529227Z","iopub.status.idle":"2022-08-15T14:37:40.892702Z","shell.execute_reply.started":"2022-08-15T14:37:40.529194Z","shell.execute_reply":"2022-08-15T14:37:40.891431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And there you go! \n\nYou can see that the target class has now a decreasing default distrbution - It is decrasing from left to right. This method is especially useful for regression problems and it also maps the categories in absolute numbers to each other! Thus, WoE transformation is not only sorting the categories, no. It also relates categorial features numerically to each other!!!","metadata":{}},{"cell_type":"code","source":"sns.displot(data, x='D_68_first', bins=10, height=5, legend=False, hue=data.target, multiple=\"stack\", aspect=2.2)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-15T14:38:16.737188Z","iopub.execute_input":"2022-08-15T14:38:16.737673Z","iopub.status.idle":"2022-08-15T14:38:17.476898Z","shell.execute_reply.started":"2022-08-15T14:38:16.737633Z","shell.execute_reply":"2022-08-15T14:38:17.475963Z"},"trusted":true},"execution_count":null,"outputs":[]}]}