{"cells":[{"metadata":{"_uuid":"3451d741362a5cb6f46fe78fd3b55569617a11bc","_cell_guid":"02271f68-2172-461c-8045-c1cf14512b45"},"cell_type":"markdown","source":"# Imbalanced binary data - Let's Downsample the majority class!\n* The data is extremely imbalanced (0.17% positives).\n* Data is also quite large, and anonymized (so we can't get many meaningful aggregations from the anonymized variables).\n* We'll downsample/negative sample by class.\n* AUC doesn't care about class balance, only seperation!\n* This will let us work with a much smaller, faster dataset for nice fast iterations :) \n\n    * NOTE! This sort of approach won't always work. However, when the minority class is far more interesting, and there's this much data, and it's so imbalanced, etc', then this sort of downsampling is an excellent appraoch.\n        * If we cared about logloss (or had some other evaluation metric, such as TPR/FPR etc'), then we could alwyas weight our data/sample/ correct the evaluation"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":false,"collapsed":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport dask\nimport os\nprint(os.listdir(\"../input\"))\n\n\n# pd.merge(pd.read_csv('../input/test.csv', dtype=dtypes),\n# pd.read_csv('../input/train.csv', dtype=dtypes).groupby(['app','channel'])['is_attributed'].mean().reset_index(),\n# on=['app','channel'], how='left').fillna(0)[['click_id','is_attributed']].to_csv('submean_app.csv', index=False)\n# # Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"dtypes = {\n        'ip':'uint32',\n        'app': 'uint16',\n        'device': 'uint16',\n        'os': 'uint16',\n        'channel': 'uint16',\n        'is_attributed': 'uint8'\n        }","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee98e138d853493a444a87d28d1b14145935cfa3","_cell_guid":"712fad3d-e0e7-4fab-a5ec-cebab436feb3"},"cell_type":"markdown","source":"## Read in train data\n* We  read it all at once then split into 2 dataframes, rather than iterating over the file twice. Note that this will mean an extra copy of the data in our RAM. \n* Dask (or maybe ray?) could speed this up nicely:\n    * https://stackoverflow.com/questions/34173859/filtering-a-large-dataframe-in-pandas-using-multiprocessing "},{"metadata":{"_uuid":"1b753a6d51a71f728f5ca73d50472b48448c19e8","scrolled":true,"_cell_guid":"bae0bc4b-5387-46a2-876d-66530f924560","trusted":false,"collapsed":true},"cell_type":"code","source":"# df_train = pd.read_csv('../input/train.csv', nrows=10000)\n# df_test = pd.read_csv('../input/test.csv', nrows=10000)\n# df_train.head()\n# df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"52afb6eb98976325272d7b78e95fb1452ee6a6ab","_cell_guid":"ccca48da-387d-4287-8507-60c3580c5d55"},"cell_type":"markdown","source":"##### Helper function to get mean values, and apply them also on test\n* We could do this more elgantly using pandas's .transform(\"mean\") , but then we'd need to merge it onto the test, and that's more annoying. \n* Note that this can easily overfit! (Better to use smoothing, e.g. bayesian average).\n* Note also that we can get feature crosses with this. (e.g. \"app=iphone AND ip=192.168\")"},{"metadata":{"_uuid":"d38e74a774c05cec88ecf8baf4e2c7281eb384c7","collapsed":true,"_cell_guid":"c1beb756-7e07-4f98-9ecb-aac15ad43938","trusted":false},"cell_type":"code","source":"# https://www.kaggle.com/cttsai/blend-app-channel-and-app-mean\n\ndef mean_feat(train, test, attrs=[]):\n    return pd.merge(test, train.groupby(attrs)['is_attributed'].mean().reset_index(), on=attrs, how='left').fillna(0).set_index('click_id')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40f2ccfa4114c56268c7377ab0ab28310e25ad81","_cell_guid":"b7c15e5f-727a-43e2-b3bf-82b4950bef42","trusted":false,"collapsed":true},"cell_type":"code","source":"import dask.dataframe as dd\ndask_df = dd.read_csv('../input/train.csv',dtype=dtypes)\ndask_df.npartitions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e61522c7326ed7e62175ab5f5f3b88e217d1e153","_cell_guid":"f678effc-d44b-453b-9f12-907dc930d503","trusted":false,"collapsed":true},"cell_type":"code","source":"df_pos = dask_df[(dask_df['is_attributed'] == 1)].compute()\nprint(\"Total positives : \",df_pos.shape[0])\ndf_neg = dask_df[(dask_df['is_attributed'] == 0)].compute()\nprint(\"Total Negatives : \",df_neg.shape[0])\nprint(\"Base percentage of positives : \",100*df_pos.shape[0]/df_neg.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"783a6a8b10f91884b090ec8e6a5bfed0a706c516","collapsed":true,"_cell_guid":"01fa60ec-4194-4052-bf2b-0e62d340309f","trusted":false},"cell_type":"code","source":"df_neg = df_neg.sample(n=3000000) # 2.25 million = 20% , 4.5 = ~10%","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c451172337b76164692f14adeb87b4c28482cf9","_cell_guid":"e03a91b4-8efe-4a52-898e-fb66339e013b","trusted":false,"collapsed":true},"cell_type":"code","source":"# join downsampled data and shuffle them\ndf = pd.concat([df_pos,df_neg]).sample(frac=1)\nprint(df.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"301aaf24876da26d8362e96e2e96ba1147b2b9eb","scrolled":true,"_cell_guid":"474b0747-6e2a-4f48-8189-bc9695f68c6a","trusted":false,"collapsed":true},"cell_type":"code","source":"df.to_csv(\"train_downsampled_3m.csv.gz\",index=False,compression=\"gzip\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28b174c850e7f0348e24676f2656e600b3a174e8","_cell_guid":"5854f3d5-54be-44f8-b431-3777d7f09a7f"},"cell_type":"markdown","source":"# As additional, prior steps we could calculate out aggregate features BEFORE the downsampling.\n* Likely to be usefu only for long tail, such as pairwise features."},{"metadata":{"_uuid":"364f25f49eb971687ac98a93fd56d6b4feb2f440","_cell_guid":"9cf92cbe-3c9c-443e-8f26-beedcbb6c72b","trusted":false,"collapsed":true},"cell_type":"code","source":"# # df = pd.concat([df_pos.sample(n=50000),df_neg.sample(n=100000)])\n# df = pd.concat([df_pos.head(n=50000),df_neg.head(n=100000)])\n# df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6bf9bb00b7ef3f9537b8fc08d17a3c60a47dc30a","collapsed":true,"_cell_guid":"34e55d70-9305-4db4-ab09-0dd14602ee22","trusted":false},"cell_type":"code","source":"## code gives error? \n\ndef group_mean(df,categoricals, target):\n\n\tfor col in categoricals:\n\n\t\tgrouped = df[target,col].groupby([col])\n\t\tn = grouped[target].transform('count')\n\t\tmean = grouped[target].transform('mean')\n\t\tdf['%s_result'%(col)] = (mean*n - df[target])/(n-1)\n\treturn df\n# group_mean(df,categoricals=[\"app\",\"os\"], target=\"is_attributed\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4a8a192969cebd614fd2861cc0a39b90b31652a","_cell_guid":"025ee8af-43b2-410f-874e-ce5ca69a9cf0","trusted":false,"collapsed":true},"cell_type":"code","source":"# group_mean(df,categoricals=[\"app\",\"os\"], target=\"is_attributed\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}