{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MultiOutput MLP - Weighted loss\n\n## References\n\nThis code uses the dataset [Indoor Unified Wifi](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids). Some code snippets are inspired by [MLP by Keras with Unified Wi-Fi Feats](https://www.kaggle.com/jerrymark611/mlp-by-keras-with-unified-wi-fi-feats). Thank you for your nice contributions.\n\n## The code\n\nThe code proposes an architecture based on shared and specialized layers for each output. The network is optimized using custom loss that weights the RMSE of (x,y) predictions and the accuracy of floor predictions.\n\n\n### Libraries and custom regression metric\n\nIn the first cell we define a custom metric that just computes the root mean squared error of our predictions.","metadata":{}},{"cell_type":"code","source":"import argparse\nimport pickle\nimport random\nimport os\n\nimport keras\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import GroupKFold\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\nfrom tensorflow_addons.layers import WeightNormalization\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nimport tensorflow.keras.backend as K\nfrom tensorflow.keras import metrics\n\ndef root_mean_squared_error(y_true, y_pred):\n    return K.sqrt(K.mean(K.square(y_pred - y_true)))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data reading","metadata":{}},{"cell_type":"code","source":"feature_dir = \"../input/indoorunifiedwifids\"\ndata = pd.read_csv(feature_dir + '/train_all.csv')\ndf_submission = pd.read_csv('../input/simple-99-accurate-floor-model/submission.csv')\ndf_submission.set_index('site_path_timestamp', inplace=True)\ndf_test = pd.read_csv(feature_dir + '/test_all.csv')\ndf_test.rename({'site_path_timestamp': 'site_path_timestamp_2'}, axis=1, inplace=True)\ndf_test.set_index('site_path_timestamp_2', inplace=True)\ndf_test.loc[df_submission.index, 'floor'] = df_submission.floor","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Most basic parameters\n\nPlay with the number of epochs and the batch size in order to improve the results!","metadata":{}},{"cell_type":"code","source":"# Most basic parameters to tune\nN_FOLDS = 5\nEPOCHS = 100\nNUM_FEATS = 100\nBATCH_SIZE = 1400","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Categorical encoding\n\nThis part of the code encodes the wifi bssids, the sites and the floors (targets). In contrast to [MLP by Keras with Unified Wi-Fi Feats](https://www.kaggle.com/jerrymark611/mlp-by-keras-with-unified-wi-fi-feats) we do not scale the RRSSID values.","metadata":{}},{"cell_type":"code","source":"BSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS = [f'rssi_{i}' for i in range(NUM_FEATS)]\n\nwifi_bssids = []\nfor i in range(100):\n    wifi_bssids.extend(data.iloc[:, i].values.tolist())\n    wifi_bssids_test = []\nfor i in range(100):\n    wifi_bssids_test.extend(df_test.iloc[:, i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\nwifi_bssids = list(set(wifi_bssids))\nwifi_bssids.extend(wifi_bssids_test)\n\n# Encoders\nle = LabelEncoder()\nle.fit(wifi_bssids)\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\nfloors_onehot = pd.get_dummies(data['floor'])\nfor i in BSSID_FEATS:\n    data.loc[:, i] = le.transform(data.loc[:, i])\n    df_test.loc[:, i] = le.transform(df_test.loc[:, i])\ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\ndf_test.loc[:, 'site_id'] = le_site.transform(df_test.loc[:, 'site_id'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training and inference\n\nFor simplicity the training and inference code is provided together. The contribution of this part of the code is the addition of shared and specialized layers in the architecture. Also, a weighted loss is provided in order to target better the competition metric.","metadata":{}},{"cell_type":"code","source":"preds_x = np.zeros(df_test.shape[0])\npreds_y = np.zeros(df_test.shape[0])\nfloor_preds = np.zeros((df_submission.shape[0], 11))\nkf = GroupKFold(n_splits=N_FOLDS)\nfor fold, (trn_idx, val_idx) in enumerate(kf.split(data, groups=data.loc[:, 'path'])):\n    train, floors_train = data.iloc[trn_idx], floors_onehot.iloc[trn_idx]\n    val, floors_val = data.iloc[val_idx], floors_onehot.iloc[val_idx]\n    train_x, train_y, train_f = train['x'], train['y'], floors_train\n    val_x, val_y, val_f = val['x'], val['y'], floors_val\n\n    # NN definition\n\n    # Embedding BSSIDS\n    input_embd_layer = L.Input(shape=(NUM_FEATS,))\n    x1 = L.Embedding(len(wifi_bssids), 64)(input_embd_layer)\n    x1 = L.Flatten()(x1)\n\n    input_layer = L.Input(NUM_FEATS, )\n\n    x2 = L.BatchNormalization()(input_layer)\n    x2 = L.Dense(NUM_FEATS * 64, activation='selu')(x2)\n    x2 = L.LeakyReLU()(x2)\n\n    # site\n    input_site_layer = L.Input(shape=(1,))\n    x3 = L.Embedding(len(data['site_id'].unique()), 8)(input_site_layer)\n    x3 = L.Flatten()(x3)\n\n    x = L.Concatenate(axis=1)([x1, x2, x3])\n\n    x = L.BatchNormalization()(x)\n    x = L.Dropout(0.3)(x)\n    x = WeightNormalization(L.Dense(512, activation='selu'))(x)\n\n    x = L.BatchNormalization()(x)\n    x = L.Dropout(0.3)(x)\n    x_shared = WeightNormalization(L.Dense(256, activation='selu'))(x)\n\n    x = L.BatchNormalization()(x_shared)\n    x = L.Dropout(0.3)(x)\n    x = WeightNormalization(L.Dense(128, activation='selu'))(x)\n\n    x = L.BatchNormalization()(x)\n    x = L.Dropout(0.3)(x)\n    x = WeightNormalization(L.Dense(64, activation='selu'))(x)\n\n    y = L.BatchNormalization()(x_shared)\n    y = L.Dropout(0.3)(y)\n    y = WeightNormalization(L.Dense(128, activation='selu'))(y)\n\n    y = L.BatchNormalization()(y)\n    y = L.Dropout(0.3)(y)\n    y = WeightNormalization(L.Dense(64, activation='selu'))(y)\n\n    f = L.BatchNormalization()(x_shared)\n    f = L.Dropout(0.3)(f)\n    f = WeightNormalization(L.Dense(128, activation='selu'))(f)\n\n    f = L.BatchNormalization()(f)\n    f = L.Dropout(0.3)(f)\n    f = WeightNormalization(L.Dense(64, activation='selu'))(f)\n\n    output_layer_x = L.Dense(1, name='output_x')(x)\n    output_layer_y = L.Dense(1, name='output_y')(y)\n    output_layer_f = L.Dense(11, activation='softmax', name='output_f')(f)\n    model = M.Model([input_embd_layer, input_layer, input_site_layer],\n                    [output_layer_x, output_layer_y, output_layer_f])\n\n    model.compile(optimizer=tf.optimizers.Adam(lr=0.005),\n                  loss={'output_x': root_mean_squared_error,\n                        'output_y': root_mean_squared_error,\n                        'output_f': 'categorical_crossentropy'},\n                  loss_weights={'output_x': 0.5, 'output_y': 0.5, 'output_f': 15},\n                  metrics={'output_x': metrics.RootMeanSquaredError(),\n                           'output_y': metrics.RootMeanSquaredError(),\n                           'output_f': metrics.CategoricalAccuracy()})\n    callback = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\", patience=35)\n    model.fit(x=[train[BSSID_FEATS], train[RSSI_FEATS], train['site_id']],\n              y=[train_x, train_y, train_f], shuffle=True, use_multiprocessing=False,\n              validation_data=([val[BSSID_FEATS], val[RSSI_FEATS], val['site_id']],\n                               [val_x, val_y, val_f]), batch_size=1600, epochs=EPOCHS,\n              callbacks=[callback], verbose=True)\n    keras.backend.clear_session()\n    preds = model.predict([df_test[BSSID_FEATS], df_test[RSSI_FEATS], df_test['site_id']])\n    pr_x, pr_y = preds[0], preds[1]\n    preds_x += pr_x.reshape(pr_x.shape[0]) / N_FOLDS\n    preds_y += pr_y.reshape(pr_x.shape[0]) / N_FOLDS\n    floor_preds += preds[2]\nfinal_preds = np.argmax(floor_preds, axis=1) - 2\ndf_test['x'] = preds_x\ndf_test['y'] = preds_y\ndf_test['floor'] = final_preds\ndf_submission.update(df_test)\ndf_submission.to_csv('submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Results\n\nAs reported in the metrics the floor accuracy in the validation set is >99.6% with no post-process at all.\n\nPublic LB: \n\nI leave the post-processing for the reader.\nSome great contributions that can boost up significatively the results are:\n* [Indoor Navigation - \"Snap to Grid\" Post Processing](https://www.kaggle.com/robikscube/indoor-navigation-snap-to-grid-post-processing)\n* [Postprocessing based on leakage](https://www.kaggle.com/tomooinubushi/postprocessing-based-on-leakage)\n* [indoor - Post-processing by Cost Minimization](https://www.kaggle.com/saitodevel01/indoor-post-processing-by-cost-minimization)\n* [with magn - Cost Minimization](https://www.kaggle.com/museas/with-magn-cost-minimization)\n\nThank you for reading and good luck!\n","metadata":{}}]}