{"cells":[{"metadata":{"_cell_guid":"f7d2ae9e-6e8a-4d20-9bcb-c5b49f28c7a0","_uuid":"7467ecdc0a1cf698acde96bb83d1ca4b0d89d791"},"cell_type":"markdown","source":"**Objective:** Predict whether a user will download an app after clicking a mobile app advertisement.\n\n**Data:** \n- `ip`: ip address of click.\n- `app`: app id for marketing.\n- `device`: device type id of user mobile phone (e.g., iphone 6 plus, iphone 7, huawei mate 7, etc.)\n- `os`: os version id of user mobile phone\n- `channel`: channel id of mobile ad publisher\n- `click_time`: timestamp of click (UTC)\n- `attributed_time`: if user download the app for after clicking an ad, this is the time of the app download\n- `is_attributed`: the target that is to be predicted, indicating the app was downloaded\n\nVariables are encoded by a numerical index. \n\n**Class Imbalance:**\n\nI won't go through an extensive EDA in this notebook, but it's worth pointing out that this dataset is highly imbalanced. Observations from the positive class make up roughly 0.25% of the dataset. Accuracy won't be a useful evaluation metric for this dataset, as a model which always predicts 0 would have an accuracy of 99.75%. You can read more about considerations for imbalanced datasets [here](https://www.jeremyjordan.me/imbalanced-data/) (a recent post from my personal blog). \n\nThe competition will be evaluated based on the AUC-ROC score (which I discuss in my blog post). \n\n**Plan**:\n\nDevelop an embedding matrix for the provided categories:\n- app id\n- device id\n- os id\n- channel id\n\nExtract date attributes from `click_time`:\n- day of week\n- hour of day\n- minute of hour\n- second of minute\n\nAssume IP represents an identity (individual or company). Group the data (click observations) by IP address and summarize activity.\n\n- Number of clicks within same minute\n- Number of clicks within same hour\n- Number of clicks within same day\n- Number of device-os pairs used within same day\n- Number of app-channel pairs used within same day\n- Number of app-os pairs used within same day\n\n---\n\n**Note:**\nThis isn't the exact notebook I used for my competition submission, but the processing steps and model configuration are largely the same. To train the model I used for submitting predictions, I used a c5.18xlarge instance on AWS (144 GB of RAM) so that I could hold the entire training dataset (~200,000,000 observations) in memory and perform the necessary feature engineering steps without worrying about memory constraints."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false,"collapsed":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sn\n\nimport numpy as np\nimport pandas as pd\npd.options.display.max_columns = 999\n\ndata_dir = '../input'\nimport os\nprint(os.listdir(data_dir))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8c059767-5fff-48eb-b95b-ec5f581cce7d","_uuid":"3b7a6ea38b760da9a886466577bc46fcd94e0a35"},"cell_type":"markdown","source":"For this notebook, we'll use the first 10 million observations in `train.csv` and further split it into training and validation sets. This sample should be small enough to use on most computers without memory concerns, although you can always decrease `nrows` if necessary."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false,"collapsed":true},"cell_type":"code","source":"# save space by using ints where possible\ndtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        'click_id'      : 'uint32'\n        }\n\ndf = pd.read_csv('../input/train.csv', nrows=10000000, dtype=dtypes, parse_dates=['click_time'])\ndf = df.drop('attributed_time', axis=1)\n\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ad78dc2b-9064-4f2f-a6f7-94515ae5135d","_uuid":"c75d80487d20f3008d98d0dd1d2edcc32d90dc72","trusted":false,"collapsed":true},"cell_type":"code","source":"df['is_attributed'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"3cfdcb7c-d473-41b3-a435-edf850f08d61","_uuid":"abd4ecbf59eb23c7a052796c69395ba2fa8ae29e","trusted":false},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(df.drop('is_attributed', axis=1), \n                                                    df['is_attributed'], test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"bb68b106-6d02-4493-acaf-a33c6401056a","_uuid":"74e6a017caa2c99a7ec6deaa78075939e068f2b5"},"cell_type":"markdown","source":"**----- Feature Engineering and Scaling ------**"},{"metadata":{"collapsed":true,"_cell_guid":"53393b22-b043-4145-b6b3-e71b827b7f04","_uuid":"34f6c84e82af384d720f0dea0723d9fa1d1c1862"},"cell_type":"markdown","source":"**Date features:**\n\n- day of week\n- hour of day\n- minute of hour\n- second of minute\n\nUnfortunately, the test set contains clicks for a different day of week than is present in the train set. I probably won't use the \"day of week\" feature in my final model as it will likely degrade the results (we won't be able to update the randomly initialized embeddings for the days we haven't seen during training). However, this value is still useful in grouping observations by day as done in the next section."},{"metadata":{"collapsed":true,"_cell_guid":"429bf735-fe1d-4895-8573-f8081dd43e4f","_uuid":"f32eda6b94baf0e4b067686f0448dd3cec13e9f5","trusted":false},"cell_type":"code","source":"def add_dateparts(df):\n    print('Adding date parts...')\n    df['day'] = df['click_time'].dt.dayofweek\n    df['hour'] = df['click_time'].dt.hour\n    df['minute'] = df['click_time'].dt.minute\n    df['second'] = df['click_time'].dt.second\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7a4a8c7e-895b-40e9-8f11-d28a833ac633","_uuid":"536edcde6714283d40fac0180b2a6c16237dd955"},"cell_type":"markdown","source":"TalkingData would like to reduce fraudulent clicks coming from click farms. \n\n![](https://cdn.techinasia.com/wp-content/uploads/2015/02/app-store-ranking-manipulation-farm.jpg)\n\nWe'll use the following features to summarize the activity coming from a specific IP. "},{"metadata":{"_cell_guid":"26b8d28f-1e5e-4e24-bb73-7beb2a0b0d01","_uuid":"e82238730cf7d740a965b9f9e41ef05fe5a0b85f"},"cell_type":"markdown","source":"**IP features: **\n\n- Number of clicks within same minute\n- Number of clicks within same hour\n- Number of clicks within same day\n- Number of device-os pairs used within same day\n- Number of app-channel pairs used within same day\n- Number of app-os pairs used within same day\n"},{"metadata":{"collapsed":true,"_cell_guid":"07442aa9-da5b-42d1-834f-5977aaf255d1","_uuid":"a8e5de9258ded27bdafb181532af4833fbde3efe","_kg_hide-input":false,"_kg_hide-output":false,"trusted":false},"cell_type":"code","source":"def add_ip_stats(df):\n    GROUPBY_AGGREGATIONS = [\n        # Click count within the minute of observation for same IP\n        {'groupby': ['ip','day','hour', 'minute'], 'select': 'minute', \n         'agg': 'count', 'feature_name': 'clicks_count_same_minute'},\n\n        # Click count within the hour of observation for same IP\n        {'groupby': ['ip','day', 'hour'], 'select': 'hour', \n         'agg': 'count', 'feature_name': 'clicks_count_same_hour'},\n\n        # Click count within the day of observation for same IP\n        {'groupby': ['ip', 'day'], 'select': 'day', \n         'agg': 'count', 'feature_name': 'clicks_count_same_day'},\n        \n        # Number of device-os pairs used within the day of observation for same IP\n        {'groupby': ['ip', 'day', 'device'], 'select': 'os', \n         'agg': 'count', 'feature_name': 'device_os_count_same_day'},\n        \n        # Number of app-channel pairs used within the day of observation for same IP\n        {'groupby': ['ip', 'day', 'app'], 'select': 'channel', \n         'agg': 'count', 'feature_name': 'app_channel_count_same_day'},\n        \n        # Number of app-os pairs used within the day of observation for same IP\n        {'groupby': ['ip', 'day', 'app'], 'select': 'os', \n         'agg': 'count', 'feature_name': 'app_os_count_same_day'}\n    ]\n\n    # Apply all the groupby transformations\n    for spec in GROUPBY_AGGREGATIONS:\n        print(f\"Grouping by {spec['groupby']}, and aggregating {spec['select']} with {spec['agg']}\")\n\n        # Unique list of features to select\n        all_features = list(set(spec['groupby'] + [spec['select']]))\n\n        # Perform the groupby\n        result = df[all_features] \\\n                    .groupby(spec['groupby'])[spec['select']] \\\n                    .agg(spec['agg']) \\\n                    .rename(spec['feature_name']) \\\n                    .reset_index()\n\n        # Merge back to X_train\n        df = df.merge(result, on=spec['groupby'], how='left')\n        \n    return df","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"0f824ee4-c344-4254-bd72-db2f3b89ada9","_uuid":"a26db76f47e8a22ce0b7d63ef424a3e7fda724cd","trusted":false},"cell_type":"code","source":"categorical_vars = ['app', 'device', 'os', 'channel', 'day', 'hour', 'minute', 'second']\nnumerical_vars = ['clicks_count_same_minute', 'clicks_count_same_hour', 'clicks_count_same_day', \n                  'device_os_count_same_day', 'app_channel_count_same_day','app_os_count_same_day']\ndrop_columns = ['ip', 'click_time']\ntarget = 'is_attributed'","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"7796ceb5-4661-4eee-a50a-58fc08ff6a15","_uuid":"64ba33cfd7c70609678e589d33a0ea6fbbdb6b08","trusted":false},"cell_type":"code","source":"def feature_pipeline(X):\n    X = add_dateparts(X)\n    X = add_ip_stats(X)\n    X = X.fillna(0)\n    X = X.drop(drop_columns, axis=1)\n    \n    return X","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2d6d5031-59dd-4ea6-b71b-6b5ed16f83bc","_uuid":"5abe150ad271cc52cea9bd27fff09703ca11d258"},"cell_type":"markdown","source":"Before feature scaling."},{"metadata":{"_cell_guid":"953628a4-cb69-477f-9837-1d4d0e5d617b","_uuid":"541b9dd3333b286f24914c046be31d2e2dcb9caf","trusted":false,"collapsed":true},"cell_type":"code","source":"feature_pipeline(X_train).head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0b3a4cfc-c549-4d8e-b2d9-0df05bf93cc9","_uuid":"032f8739b6c1344fbdaab5c2890e39d4c9a84d3a"},"cell_type":"markdown","source":"After adding the date and IP activity features, we should scale the numerical variables to have zero mean and unit variance. This will help ensure a smoother learning process for the network. "},{"metadata":{"_cell_guid":"40205b65-eae8-4dda-a541-d1b703c41128","_uuid":"c891ad7e9e46b1f650aa3354519b00c90f40eca3","trusted":false,"collapsed":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nscaler = StandardScaler()\n\nX_train = feature_pipeline(X_train)\nX_train[numerical_vars] = scaler.fit_transform(X_train[numerical_vars])\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0cb0946c-5a36-4ad7-9f50-97ad46afaaa9","_uuid":"3732cab35198cc24ed770614fd164dadceff395b"},"cell_type":"markdown","source":"Before feature scaling."},{"metadata":{"_cell_guid":"34b5b0c5-2a8f-42e7-96cc-1a5400da7ad7","_uuid":"05af46dbb0ebc62b032abcf11f6bc7b64cfd465c","trusted":false,"collapsed":true},"cell_type":"code","source":"feature_pipeline(X_test).head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fc417a44-6432-461e-bd63-7b0f17f1006b","_uuid":"1404cf77daee2d4532210ac1b21b0223e986ddb6"},"cell_type":"markdown","source":"After feature scaling."},{"metadata":{"_cell_guid":"b20ce213-998e-4687-8529-6b75fffff695","_uuid":"626e00d4a1cbfd069aba2b5bb9f80330d295c925","trusted":false,"collapsed":true},"cell_type":"code","source":"X_test = feature_pipeline(X_test)\nX_test[numerical_vars] = scaler.transform(X_test[numerical_vars])\nX_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a1443d6d-d609-4c9a-9348-916ba438f1ad","_uuid":"c1066c62fec14d53118ef3ba5de9ec543f59e161"},"cell_type":"markdown","source":"Our model is going to have a number of inputs, so we'll prepare a dictionary of input values to feed into the model. The dictionary key should match the layer name of the intended input. "},{"metadata":{"collapsed":true,"_cell_guid":"05d13e53-ba68-4b57-89ff-8ece8fc67dd5","_uuid":"3d6ac6b99c0a01078cf5a764b21cb812b5fd96e6","trusted":false},"cell_type":"code","source":"def get_keras_data(dataset):\n    X = {\n        'app': dataset['app'],\n        'dev': dataset['device'],\n        'os': dataset['os'],\n        'ch': dataset['channel'],\n        \n        'h': dataset['hour'],\n        'm': dataset['minute'],\n        's': dataset['second'],\n        \n        'ip_stats': dataset[numerical_vars]\n    }\n    return X\n\ntrain_data = get_keras_data(X_train)\nval_data = get_keras_data(X_test)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"697773dd-576c-4019-86a5-655b59dcef72","_uuid":"a82e3ef1f9630c9383696676d07e977f56df04e0"},"cell_type":"markdown","source":"**----- Model Definition ------**"},{"metadata":{"_cell_guid":"c52367fd-474f-4002-ae2a-af6dbe27043e","_uuid":"f3be4c1ebf8c37614cb8c8aaddba5076b8ec480d"},"cell_type":"markdown","source":"Rather than one-hot encoding the categorical features, we're going to learn a set of features to describe each category. We'll do this with an embedding layer for *each* categorical input. This embedding layer will essentially act as a lookup table where each instance of a class is described by $n$ parameters which can be learned (via backprop) during training. \n\n![](https://i.imgur.com/CADVONq.png)\n\nIt's important to note that we initialize the embedding matrix with random values, so category instances which weren't seen during training will still be described by random values and should not be trusted.\n\nYou can read more about entity embedding here:\n- [Entity Embeddings of Categorical Variables](https://arxiv.org/abs/1604.06737)\n- [Artificial Neural Networks Applied to Taxi Destination Prediction](https://arxiv.org/abs/1508.00021)"},{"metadata":{"_cell_guid":"877627b8-2b4e-4f4a-b4d5-d4bf2177a365","_uuid":"d940b7756262b50d42eaa2316d12a1d1800a0ff9"},"cell_type":"markdown","source":"The model architecture is visualized below. \n\n![](https://i.imgur.com/2t5P0vu.png)\n\n**Edit:** I found that performance was just as good with one dense layer, so I opted to go for the simpler model. \n\n**Note:** If you decide to build a deeper model, remember to add [BatchNorm](https://www.jeremyjordan.me/batch-normalization/) to your layers to improve the model's learning. "},{"metadata":{"_cell_guid":"993efe07-a035-43f1-8602-9efce1deb5e6","_uuid":"77c6799600c20488c06d168c55381ad621f0d372","trusted":false,"collapsed":true},"cell_type":"code","source":"from keras.models import Sequential, Model\nfrom keras.layers import Input, Embedding, concatenate, Dense, Dropout, Lambda\nfrom keras import regularizers\nimport keras.backend as K\n\nmax_app = np.max([X_train['app'].max(), X_test['app'].max()])+1\nmax_dev = np.max([X_train['device'].max(), X_test['device'].max()])+1\nmax_os = np.max([X_train['os'].max(), X_test['os'].max()])+1\nmax_ch = np.max([X_train['channel'].max(), X_test['channel'].max()])+1\n\nmax_d = 7\nmax_h = 24\nmax_m = 60\nmax_s = 60\n\nemb_dims = (max_app, max_dev, max_os, max_ch, max_d, max_h, max_m, max_s)\n\ndef build_model(emb_dims, optimizer='adam'):\n    # ------ Embeddings ------\n    max_app, max_dev, max_os, max_ch, max_d, max_h, max_m, max_s = emb_dims\n    emb_cat_n = 20\n    emb_time_n = 5\n\n    in_app = Input(shape=[1], name = 'app')\n    emb_app = Embedding(max_app, emb_cat_n)(in_app)\n\n    in_dev = Input(shape=[1], name = 'dev')\n    emb_dev = Embedding(max_dev, emb_cat_n)(in_dev)\n\n    in_os = Input(shape=[1], name = 'os')\n    emb_os = Embedding(max_os, emb_cat_n)(in_os)\n\n    in_ch = Input(shape=[1], name = 'ch')\n    emb_ch = Embedding(max_ch, emb_cat_n)(in_ch)\n\n#   ---- if we were using this model in production we'd probably add this back in and train over a larger timeframe ----\n#   in_d = Input(shape=[1], name = 'd')\n#   emb_d = Embedding(max_d, emb_time_n)(in_d) \n\n    in_h = Input(shape=[1], name = 'h')\n    emb_h = Embedding(max_h, emb_time_n)(in_h) \n\n    in_m = Input(shape=[1], name = 'm')\n    emb_m = Embedding(max_m, emb_time_n)(in_m) \n\n    in_s = Input(shape=[1], name = 's')\n    emb_s = Embedding(max_s, emb_time_n)(in_s) \n\n\n    embeddings = concatenate([emb_app, emb_dev, emb_os, emb_ch, emb_h, emb_m, emb_s])\n    embeddings_squeeze = Lambda(lambda x: K.squeeze(x, axis=1))(embeddings) # remove sequential dimension\n\n    # ------ Numerical Inputs ------\n    in_ip_stats = Input(shape=[len(numerical_vars)], name = 'ip_stats')\n\n    # ------ Model ------\n    dense_n = 600\n    \n    X = concatenate([embeddings_squeeze, in_ip_stats])\n    X = Dense(dense_n, activation='relu', kernel_regularizer=regularizers.l2(0.01))(X)\n    X = Dropout(0.8)(X)\n    pred = Dense(1, activation='sigmoid')(X)\n\n    model = Model(inputs=[in_app, in_dev, in_os, in_ch, in_h, in_m, in_s,\n                          in_ip_stats], \n                  outputs=pred)\n\n    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"2218034b-8b0e-41ab-8c03-d90a90524c56","_uuid":"775f5d0ef569c5cb2c17fc01b0280933ff074770","trusted":false},"cell_type":"code","source":"model = build_model(emb_dims)\n# model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9e93da34-7f75-482e-a89d-8157e9841186","_uuid":"6f77da3003179b70182aef4c7af22dc8ce2c3043"},"cell_type":"markdown","source":"**----- Model Training ------**"},{"metadata":{"_cell_guid":"d44188b0-42fd-45e5-8ecd-e8269af42979","_uuid":"d931cfd528a4951100d703e00b78911a53ed09f5"},"cell_type":"markdown","source":"Let's create a few callbacks to use during training. "},{"metadata":{"collapsed":true,"_cell_guid":"2c3210f7-d28a-44cb-a3db-0a33e4078a21","_uuid":"39c506ba03bd0f40b9bbeeb9c290de331cb2a276","trusted":false},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint\nfrom keras.callbacks import LearningRateScheduler\n\ncheckpoint = ModelCheckpoint('best_weights.h5', monitor='val_loss', verbose=0, save_best_only=True)\n\ndef step_decay_schedule(initial_lr=1e-3, decay_factor=0.75, step_size=10):\n    '''\n    Wrapper function to create a LearningRateScheduler with step decay schedule.\n    '''\n    def schedule(epoch):\n        return initial_lr * (decay_factor ** np.floor(epoch/step_size))\n    \n    return LearningRateScheduler(schedule)\n\nlr_sched = step_decay_schedule(initial_lr=1e-2, decay_factor=0.75, step_size=1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6ebd8b7a-b69f-448c-ab8e-5b7d7e504872","_uuid":"f275eec0bddae3d16e81e935a5d7041540efcb47"},"cell_type":"markdown","source":"We'll introduce a class weight such that our update step is 4x as large for observations of the positive (rare) class. "},{"metadata":{"_cell_guid":"7b137b72-ee54-4a9b-ba2e-59063f179c85","_uuid":"981afc2c072659d3bee1cd3497c3501a2c8f3e8a","trusted":false,"collapsed":true},"cell_type":"code","source":"class_weight = {0:0.20, 1:0.80}\nhistory = model.fit(train_data, y_train, batch_size=2048, epochs=3, \n                    class_weight=class_weight, validation_data=(val_data, y_test), \n                    callbacks=[checkpoint, lr_sched])","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"4742444d-8883-45c7-893c-9eb7fc465b8e","_uuid":"6f8a309cba44fc8fa28a54cc712fd0317668decb","trusted":false},"cell_type":"code","source":"model.load_weights('best_weights.h5')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"02321962-1b20-494a-83cc-6ae0f6b3e0d5","_uuid":"2ee097a9109359479234e300a58e9cae2f7b6b22"},"cell_type":"markdown","source":"Just for kicks, let's also explore what it would look like to train an ensemble of models. "},{"metadata":{"collapsed":true,"_cell_guid":"8913c28a-1d25-44c3-9335-3ec801475a00","_uuid":"2a3626e3bc4a823bad39f27db2145c9fada900c5","trusted":false},"cell_type":"code","source":"class EnsembleNN:\n    def __init__(self, model_builder, weights_paths, emb_dims):\n        self.models = []\n        for weights in weights_paths:\n            model = model_builder(emb_dims)\n            model.load_weights(weights)\n            self.models.append(model)\n            \n    def predict(self, X):\n        preds = [m.predict(X) for m in self.models]\n        final_preds = np.mean(preds, axis=0)\n        # if we really wanted to maximize performance, we'd weight our predictions instead of taking a simple mean\n        return final_preds","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c8d855f5-860f-4dc4-9375-f6be76986666","_uuid":"89970658a6caefde4e074237027166ef7b8369f8","trusted":false,"collapsed":true},"cell_type":"code","source":"class_weight = {0:0.20, 1:0.80}\n\nfor i in range(3): \n    print(f'--- Training Model {i+1} of {3} ---')\n    model = build_model(emb_dims)\n    checkpoint = ModelCheckpoint(f'best_weights_{i}.h5', monitor='val_loss', verbose=0, save_best_only=True)\n    history = model.fit(train_data, y_train, batch_size=2048, epochs=3, \n                        class_weight=class_weight, validation_data=(val_data, y_test), \n                        callbacks=[checkpoint, lr_sched])","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"196b0b30-dcb2-4c6c-bd64-86cceeb84dfa","_uuid":"e4d6f117d833b0b8baf26035aa8e643ac9f2c657","trusted":false},"cell_type":"code","source":"weights = ['best_weights_0.h5', 'best_weights_1.h5', 'best_weights_2.h5']\n\nensemble_nn = EnsembleNN(build_model, weights, emb_dims)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7f0b8501-9cdb-4d83-b616-e8241af2cce6","_uuid":"9a1fa5f028ce00e6d1cd847bb834804ea6510f65"},"cell_type":"markdown","source":"**----- Evaluation ------**"},{"metadata":{"_cell_guid":"d8a05ed0-d141-44d5-9d70-43a582e4bf6e","_uuid":"3d10a26cf2d0f8d61ddbefa1f5ca776abf79646d","trusted":false,"collapsed":true},"cell_type":"code","source":"plt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"920d6770-b5bf-464f-b79e-456158902963","_uuid":"69b202c2bfe8c6f5329065d4fa247f8adb550a8e","trusted":false,"collapsed":true},"cell_type":"code","source":"from sklearn.metrics import roc_curve, roc_auc_score\n\npreds = model.predict(val_data)\nfpr, tpr, thresholds = roc_curve(y_test, preds, pos_label=1)\n\nauc = roc_auc_score(y_test, preds)\n\nfig, ax = plt.subplots()\nax.plot(fpr, tpr)\nax.plot([0, 1], [0, 1], color='navy', linestyle='--', label='random')\nplt.title(f'Single Model AUC: {auc}')\nax.set_xlabel('False positive rate')\nax.set_ylabel('True positive rate')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d4048c87-f914-4e95-9291-e6e7b8502e73","_uuid":"d65e48f70eb790b4a787f3ddcf97087b3f936843","trusted":false,"collapsed":true},"cell_type":"code","source":"preds = ensemble_nn.predict(val_data)\nfpr, tpr, thresholds = roc_curve(y_test, preds, pos_label=1)\n\nauc = roc_auc_score(y_test, preds)\n\nfig, ax = plt.subplots()\nax.plot(fpr, tpr)\nax.plot([0, 1], [0, 1], color='navy', linestyle='--', label='random')\nplt.title(f'Ensemble Model AUC: {auc}')\nax.set_xlabel('False positive rate')\nax.set_ylabel('True positive rate')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"59ec45d3-4235-42fd-abd3-ccc2a54ab586","_uuid":"d5dadb53eee0b38315c718d5895c956c103bc8c0"},"cell_type":"markdown","source":"---\n\n**Visualize embedding layers.**\n\nThe features provided were encoded, so we can't analyze the embeddings to see whether or not the learned representation makes semantic sense (ie. various Android OS's are nearby in the embedding space). However, we can still inspect to see if various clusters have formed. "},{"metadata":{"collapsed":true,"_cell_guid":"4d4cd488-5817-4f8e-83fd-1c9712562a7b","_uuid":"f0224755453967eba66c7e076ff6f4d69ea9769c","trusted":false},"cell_type":"code","source":"from sklearn.manifold import TSNE\n\ndef show_embeddings(layer_weights, name='Embedding'):\n    tsne = TSNE(n_components=2, init='pca', random_state=0, method='exact')\n    Y = tsne.fit_transform(layer_weights)\n    plt.figure(figsize=(8,8))\n    plt.scatter(Y[:, 0], Y[:, 1])\n    plt.title(name)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"254e7c7b-ee1d-4c42-a78b-099dcde5bed1","_uuid":"51f56a37a8720b8fda94b2174c11223c7767b31d","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[7].get_weights()[0], name='App Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"0c71f365-90be-4ed9-a23d-9ed9e71f258a","_uuid":"cf3887eb2bf74060824012ea5c3639bbdb6ecf6c","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[8].get_weights()[0], name='Device Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"de5d5e1d-81c7-4a74-acc0-9abed8a17f80","_uuid":"e9f4d25ad98286eb699597074d29ee7af954acb4","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[9].get_weights()[0], name='OS Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"8c10c3d2-944c-48eb-844e-e8995e0f5a79","_uuid":"feb8d7927822f4ac6886e6b9dc48e9fe98fcbcbf","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[10].get_weights()[0], name='Channel Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"71f5e021-696c-426a-9fe3-6ad40e76859f","_uuid":"f22e39254f86c2a26d8e52b06f10c94cc4c36753","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[11].get_weights()[0], name='Hour Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"2d7354c6-a4e0-4a8f-b81f-b5564bfd1882","_uuid":"895ff7e1fbbb5972e380faeac815df3dab10ae28","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[12].get_weights()[0], name='Minute Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"0b10d2df-ee8b-4c4c-a693-82cd16db265e","_uuid":"7e9a26aece10ad5de53d87c24509fb8f257fda9f","trusted":false},"cell_type":"code","source":"show_embeddings(model.layers[13].get_weights()[0], name='Second Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"655c1988-1719-49e6-9b74-0e6dc39f6f33","_uuid":"78c92a6180e54219899267f01cbedc6614009a5b"},"cell_type":"markdown","source":"**Reflections:**\n\nIt doesn't look like the time features are too useful. I should try training a model without these features and see how the performance changes. \n\nThere's a *lot* of devices, I should consider collection the top $X$ devices and throwing the rest into an \"Other\" category. \n\nI should color code the scatter plots according to the category frequency. I wonder if the points which appear to be randomly distributed (ie. haven't been pushed to a distinct cluster) are due to the fact that these class instances are less frequently observed in the data. "},{"metadata":{"collapsed":true,"_cell_guid":"b152d737-691e-4662-8b9b-81aeb11cc0ee","_uuid":"4071c134fec6c31c18d396633af6ac349097bbe3","trusted":false},"cell_type":"code","source":"tsne = TSNE(n_components=2, init='pca', random_state=0, method='exact')\nY = tsne.fit_transform(model.layers[11].get_weights()[0])\n\nannotations = ['night'] * 7 + ['morning'] * 6 + ['afternoon'] * 6 + ['evening'] * 5\ncolors = [0] * 7 + [1] * 6 + [2] * 6 + [3] * 5\n\nplt.figure(figsize=(8,8))\nplt.scatter(Y[:, 0], Y[:, 1], c=colors)\nfor i, text in enumerate(annotations):\n    plt.annotate(text, (Y[i, 0],Y[i, 1]), xytext = (4, 2), textcoords = 'offset points')\nplt.title('Hour Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"7a16d417-4c82-4c5f-a303-56f959e3cf68","_uuid":"a5cc9d077b60dd0b2983ed76a90d4904c51d0169","trusted":false},"cell_type":"code","source":"tsne = TSNE(n_components=2, init='pca', random_state=0, method='exact')\nY = tsne.fit_transform(model.layers[12].get_weights()[0])\n\nplt.figure(figsize=(8,8))\nplt.scatter(Y[:, 0], Y[:, 1], c=range(60))\nplt.colorbar()\nplt.title('Minute Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"f8e7bbf2-8e1f-4e6e-949f-39cb6355f7be","_uuid":"9834369fc839ff97cad327f2584225ab79b0fad8","trusted":false},"cell_type":"code","source":"tsne = TSNE(n_components=2, init='pca', random_state=0, method='exact')\nY = tsne.fit_transform(model.layers[9].get_weights()[0])\n\nfrom matplotlib.colors import LogNorm\nfrom collections import Counter\n\n# merge dicts to initialize all counts to zero, then fill observed counts keeping others at 0\nfreq = {**{i: 0 for i in range(max_os)}, **Counter(X_train['os'])} \n\nplt.figure(figsize=(8,8))\nplt.scatter(Y[:, 0], Y[:, 1], c=list(freq.values()), norm=LogNorm())\nplt.colorbar()\nplt.title('OS Embeddings')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fa253d18-827d-4531-96bb-d3839e9fe55c","_uuid":"ed5a28128b5e2fa87955225899ef9892586772e6"},"cell_type":"markdown","source":"Looking at the OS embeddings with labels, we can observe that clusters have formed for class instances which have a higher frequency in our dataset. "},{"metadata":{"_cell_guid":"23860757-f6ab-4b6b-bf0d-4ce08b418499","_uuid":"bd9aece12b0675dd10e9e63cf50452aa4192bc7e"},"cell_type":"markdown","source":"---\n\n**One Hot Encoding**:\n\nLet's try seeing how well the model performs without the use of an embeddings layer. "},{"metadata":{"collapsed":true,"_cell_guid":"436bbffd-bff6-43c8-88e2-27b331a3ee67","_uuid":"db59a59a5ec2c70711335886f8f96573b99e4440","trusted":false},"cell_type":"code","source":"# first, let's clear a few things from memory that we no longer need\nimport gc\ndel df\ndel train_data\ndel val_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"f48de925-656e-45cf-b8bf-18e5114fa3ea","_uuid":"8ed7e5c7a1dd728fd5315bfa1b0d61af9004c450","trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\n\ncategorical_idx = [X_train.columns.get_loc(c) for c in X_train.columns if c in categorical_vars]\nenc = OneHotEncoder(categorical_features=categorical_idx)\n\nenc = enc.fit(pd.concat([X_train, X_test]))","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"b561f9b9-32a8-4102-b38c-cac87794f0f9","_uuid":"cc23901ae783ac3a0f128f0e429bbc927f3bc410","trusted":false},"cell_type":"code","source":"X_train_encoded = enc.transform(X_train).tocsr()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"0415d51d-925e-49e6-b160-fab343bf1919","_uuid":"714a0ddb3f13898ab8adfce6a16fa80c376bc32e","trusted":false},"cell_type":"code","source":"X_test_encoded = enc.transform(X_test).tocsr()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"fc708e75-520b-4aaa-a9bf-25286f5a2bb5","_uuid":"a01cea002225db38eadc201ea064499022c9b19e","trusted":false},"cell_type":"code","source":"X_train_encoded.shape","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"4032e9b3-d546-4e2a-97be-c959e1a4d464","_uuid":"697138a019e873b9f6f71c24536ecd5ad97b0a20","trusted":false},"cell_type":"code","source":"X_test_encoded.shape","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"aef437a1-74e2-434f-924b-8a43ebd17d0d","_uuid":"d8d15dc7e55cb48d352b0b677a03f66c4878b68a","trusted":false},"cell_type":"code","source":"from keras.models import Sequential, Model\nfrom keras.layers import Input, Embedding, concatenate, Dense, Dropout, Lambda\nfrom keras import regularizers\nimport keras.backend as K\n\ndef build_model_onehot(optimizer='adam'):\n    # ------ Numerical Inputs ------\n    input_layer = Input(shape=[X_train_encoded.shape[1]], name = 'input')\n\n    # ------ Model ------\n    X = Dense(450, activation='relu', kernel_regularizer=regularizers.l2(0.01))(input_layer)\n    X = Dropout(0.8)(X)\n    X = Dense(300, activation='relu', kernel_regularizer=regularizers.l2(0.01))(X)\n    X = Dropout(0.6)(X)\n    pred = Dense(1, activation='sigmoid')(X)\n\n    model = Model(inputs=input_layer, \n                  outputs=pred)\n\n    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"336646f8-a4a3-4056-b01b-48c90f80dfe6","_uuid":"c47ec3098a08ef8bf1c7694ed2679d0c29e7c319","trusted":false},"cell_type":"code","source":"model_onehot = build_model_onehot()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_cell_guid":"7e20ae8a-46df-4ea9-8759-f53f826526dd","_uuid":"06d15498186bc0cc42eab1f1419ab9941a894049","trusted":false},"cell_type":"code","source":"class_weight = {0:0.25,1:0.75}\nhistory = model_onehot.fit(X_train_encoded, y_train, batch_size=1024, epochs=5, \n                           class_weight=class_weight, validation_data=(X_test_encoded, y_test))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a103b602-3c4d-4cd7-8049-1f3415117c39","_uuid":"35e2131b86c94e180e246a67f618024a2361ca58"},"cell_type":"markdown","source":"Note: Training is also much slower when we use one-hot encoded features in place of embeddings!"},{"metadata":{"collapsed":true,"_cell_guid":"fd954de6-5936-4206-92a2-4fa51847662b","_uuid":"70f04b05088f3e5f59a407d623abed7f47ab28dc","trusted":false},"cell_type":"code","source":"from sklearn.metrics import roc_curve, roc_auc_score\n\npreds = model_onehot.predict(X_test_encoded)\nfpr, tpr, thresholds = roc_curve(y_test, preds, pos_label=1)\n\nauc = roc_auc_score(y_test, preds)\n\nfig, ax = plt.subplots()\nax.plot(fpr, tpr)\nax.plot([0, 1], [0, 1], color='navy', linestyle='--', label='random')\nplt.title(f'AUC: {auc}')\nax.set_xlabel('False positive rate')\nax.set_ylabel('True positive rate')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}