{"cells":[{"metadata":{"_cell_guid":"7419f5af-bb0a-45e4-ba6b-392bc3804a58","_uuid":"a5aafe151a94b4e3b6f9c966406c90208bfbb07e"},"cell_type":"markdown","source":"In this little kernel I will showcase how to use a basic Random Forest classifier to tackle the TalkingData problem. Also, as a small bonus, I will throw in some basic validation as well, plotting the AUC score of our classifier. Let's get started by reading the train and test datasets."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false,"collapsed":true},"cell_type":"code","source":"import gc\nimport time\nimport numpy as np\nimport pandas as pd\nfrom sklearn.cross_validation import train_test_split\nimport warnings\nwarnings.filterwarnings('ignore')\nimport os\nos.environ['OMP_NUM_THREADS'] = '4'\n\npath = '../input/'\ndtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        'click_id'      : 'uint32'\n        }\n\nprint('started loading')\n\ntrain_columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'is_attributed']\ntrain_df = pd.read_csv(path+\"train.csv\", dtype=dtypes, skiprows = range(1, 131886954), usecols=['ip','app','device','os', 'channel', 'click_time', 'is_attributed'])\ntest_df = pd.read_csv(path+\"test.csv\", dtype=dtypes, usecols=['ip','app','device','os', 'channel', 'click_time', 'click_id'])\n\nprint('finished loading')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f516ed313e9ad27192d12621eab6704f4c8f585"},"cell_type":"markdown","source":"The preprocessing step comes from [this excellent kernel](https://www.kaggle.com/alexanderkireev/deep-learning-support-9663)."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false,"collapsed":true},"cell_type":"code","source":"def prep_data(d):\n    print('hour, day, wday....')\n    d['hour'] = pd.to_datetime(d.click_time).dt.hour.astype('uint8')\n    d['day'] = pd.to_datetime(d.click_time).dt.day.astype('uint8')\n    d['wday']  = pd.to_datetime(d.click_time).dt.dayofweek.astype('uint8')\n    print('grouping by ip-day-hour combination....')\n    gp = d[['ip','day','hour','channel']].groupby(by=['ip','day','hour'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'qty'})\n    d = d.merge(gp, on=['ip','day','hour'], how='left')\n    del gp; gc.collect()\n    print('group by ip-app combination....')\n    gp = d[['ip','app', 'channel']].groupby(by=['ip', 'app'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'ip_app_count'})\n    d = d.merge(gp, on=['ip','app'], how='left')\n    del gp; gc.collect()\n    print('group by ip-app-os combination....')\n    gp = d[['ip','app', 'os', 'channel']].groupby(by=['ip', 'app', 'os'])[['channel']].count().reset_index().rename(index=str, columns={'channel': 'ip_app_os_count'})\n    d = d.merge(gp, on=['ip','app', 'os'], how='left')\n    del gp; gc.collect()\n    print(\"vars and data type....\")\n    d['qty'] = d['qty'].astype('uint16')\n    d['ip_app_count'] = d['ip_app_count'].astype('uint16')\n    d['ip_app_os_count'] = d['ip_app_os_count'].astype('uint16')\n    print(\"label encoding....\")\n    from sklearn.preprocessing import LabelEncoder\n    d[['app','device','os', 'channel', 'hour', 'day', 'wday']].apply(LabelEncoder().fit_transform)\n    print('dropping')\n    d.drop(['click_time', 'ip'], 1, inplace=True)\n    \n    return d\n\n\ntrain_df = prep_data(train_df)\ntest_df = prep_data(test_df)\nprint(\"finished\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"febdd7f36b508cf34cbc5905cbf75b77d014e444"},"cell_type":"markdown","source":"Now we will split the training data into training and validation."},{"metadata":{"_cell_guid":"f6088b36-ceae-48cb-b6d2-32b162b0397c","collapsed":true,"_uuid":"0f63e8599f40d5d7a5e8f0bff1cc42fc1881b8bf","trusted":false},"cell_type":"code","source":"RANDOM_SEED = 1\nimport random\nrandom.seed(RANDOM_SEED)\nnp.random.seed(RANDOM_SEED)\n\nX_train, X_test = train_test_split(train_df, test_size=0.1, random_state=RANDOM_SEED)\n\ny_train = X_train['is_attributed']\nX_train = X_train.drop(['is_attributed'], axis=1)\ny_test = X_test['is_attributed']\nX_test = X_test.drop(['is_attributed'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"42cdb83c-d1d7-4020-8e00-779ead2bb8b6","_uuid":"e6eed8ccb2dd8e33de633e47a1e250362e65dec7"},"cell_type":"markdown","source":"Let's now train our Random Forest Classifier (where `max_depth` shows the maximum depth of each tree)."},{"metadata":{"_cell_guid":"7c949cbc-cc33-4e7f-9a4f-3b39f51a38a3","collapsed":true,"_uuid":"b62db2a75e01fc5a595c59675a3da009a5b8ff73","trusted":false},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nrf = RandomForestClassifier(max_depth=9, random_state=0)\nrf.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7b7e5830-652c-41cb-bfed-e0695edb2654","_uuid":"5d9f2fcebe07301a257791257b953684d3a47e45"},"cell_type":"markdown","source":"Since we do not want the classifier to predict 0 or 1, but we want to have output in the range [0, 1], we will use `predict_proba` for the predictions, which returns the probability of the item being assigned to each class."},{"metadata":{"_cell_guid":"8b15c732-8c0d-475e-be35-0f63d83d09f9","collapsed":true,"_uuid":"9777c4ea994a5516308322a404c28555fd579373","trusted":false},"cell_type":"code","source":"predictions = rf.predict_proba(X_test)\npredictions","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ae456e28-e436-4c96-8d16-d439df944e27","_uuid":"fa4cba7f936bf6cddf49e6a3924a4ae7b1a84f4f"},"cell_type":"markdown","source":"As we can see `predictions` is a 2D array of `(class1_prob, class2_prob)`, where `class1_prob = 1 - class2_prob`. This is not the desirable format. We want a single value for `is_attributed`. The first class corresponds to `0` and the second to `1`. So, we need to convert `predictions` to a proper 1D submission by subtracting from the whole 1 the first value of each probability prediction. That way we will end up with an 1D array of [0, 1] probability values."},{"metadata":{"_cell_guid":"3d3301b7-51db-45aa-9d36-68a09804e270","collapsed":true,"_uuid":"e196d4f86bb8af7659c1aa970356da547ede928c","trusted":false},"cell_type":"code","source":"def convert_preds(raw_preds):\n    preds = []\n    for p in raw_preds:\n        preds.append(1 - p[0])\n    return preds","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"86946449-9436-4690-91f6-9a7064dfc7cf","collapsed":true,"_uuid":"9a11ed0fcd4ba91d304fd8e4c53f36d1b3d6a7d8","trusted":false},"cell_type":"code","source":"val_preds = convert_preds(predictions)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0f23b5b8-6bb8-4c48-a4c9-34b2b88acf5c","collapsed":true,"_uuid":"f4edef668c055d55635ee6a0668b05f888335049","trusted":false},"cell_type":"code","source":"max(val_preds)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0841c7ef-b1b2-4aec-9194-0aba534bc3f3","_uuid":"5518301210df3f84dbec261d20e50a308c0a88d7"},"cell_type":"markdown","source":"Let's now calculate and plot the AUC score of the validation set. The code for this can be [here](https://www.kaggle.com/antmarakis/calculating-and-plotting-auc-score)."},{"metadata":{"_cell_guid":"fc360cad-ef1a-4cc9-8f6b-4a5f66c6b0bd","collapsed":true,"_uuid":"28e82d0878361b70563784e6062e877d3b86af60","trusted":false},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom sklearn.metrics import (confusion_matrix, precision_recall_curve, auc,\n                             roc_curve, recall_score, classification_report, f1_score,\n                             precision_recall_fscore_support)\n\nfpr, tpr, thresholds = roc_curve(y_test, val_preds)\nroc_auc = auc(fpr, tpr)\n\nplt.title('Receiver Operating Characteristic')\nplt.plot(fpr, tpr, label='AUC = %0.4f'% roc_auc)\nplt.legend(loc='lower right')\nplt.plot([0,1],[0,1],'r--')\nplt.xlim([-0.001, 1])\nplt.ylim([0, 1.001])\nplt.ylabel('True Positive Rate')\nplt.xlabel('False Positive Rate')\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"965ff72e-0fda-465d-9484-d102ca5ac81c","_uuid":"97a7034fcd5a0442ba2c80e2404db37ac1f03d07"},"cell_type":"markdown","source":"Finally, let's predict items from the test dataset:"},{"metadata":{"_cell_guid":"c676e4a2-7c45-4be4-8fa9-40a64fe46d15","collapsed":true,"_uuid":"055dab4b3505f0cc1f95299e61b98dff1d59e9c4","trusted":false},"cell_type":"code","source":"ids = test_df['click_id']\ntest_df.drop('click_id', axis=1, inplace=True)\npredictions = rf.predict_proba(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4cf52db2-c239-482f-9bad-2cbe484ac98c","collapsed":true,"_uuid":"fd4470db7bf84195fe36d9c9c4a65ec5857465d1","trusted":false},"cell_type":"code","source":"sub_preds = convert_preds(predictions)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f665b01d-a225-4ced-9b2f-5ac0540ea4c3","collapsed":true,"_uuid":"525461781b5267eced05620cf3adffa8a2857969","trusted":false},"cell_type":"code","source":"sub = pd.DataFrame()\nsub['click_id'] = ids\nsub['is_attributed'] = sub_preds\nsub.to_csv('rf_sub.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}