{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Overview\n\nIt demonstrats how to utilize [the unified Wi-Fi dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\nThe Neural Net model is not optimized, there's much space to improve the score. \n\nIn this notebook, I refer these two excellent notebooks.\n* [wifi features with lightgbm/KFold](https://www.kaggle.com/hiro5299834/wifi-features-with-lightgbm-kfold) by [@hiro5299834](https://www.kaggle.com/hiro5299834/)<br>\n I took some code fragments from his notebook.\n* [Simple 👌 99% Accurate Floor Model 💯](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model) by [@nigelhenry](https://www.kaggle.com/nigelhenry/)<br>\n I use his excellent work, the \"floor\" prediction.\n\nIt takes much much time to finish learning. <br>\nAnd even though I enable the GPU, it doesn't help. <br>\nIf anybody knows how to make it better, can you please make a comment? <br>\n\nThank you!","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\n\nimport random\nimport os\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nimport tensorflow.keras.backend as K\nimport tensorflow_addons as tfa\nfrom tensorflow_addons.layers import WeightNormalization\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"N_SPLITS = 10\n\nepochs = 1000\n\nbatch_size = 1280\n\nSEED = 2021\n\nNUM_FEATS = 40 \n\nbase_path = '/kaggle'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n    \ndef comp_metric(xhat, yhat, fhat, x, y, f):\n    intermediate = np.sqrt(np.power(xhat-x, 2) + np.power(yhat-y, 2)) + 15 * np.abs(fhat-f)\n    return intermediate.sum()/xhat.shape[0]\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_dir = f\"{base_path}/input/indoorunifiedwifids\"\nsubm = pd.read_csv(f'{base_path}/input/indoor-location-navigation/sample_submission.csv', index_col=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(f'{feature_dir}/train_all.pkl', 'rb') as f:\n  data = pickle.load( f)\n\nwith open(f'{feature_dir}/test_all.pkl', 'rb') as f:\n  test_data = pickle.load(f)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS  = [f'rssi_{i}' for i in range(NUM_FEATS)]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wifi_bssids = []\nfor i in range(100):\n    wifi_bssids.extend(data.iloc[:,i].values.tolist())\nwifi_bssids = list(set(wifi_bssids))\n\nwifi_bssids_size = len(wifi_bssids)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids_test = []\nfor i in range(100):\n    wifi_bssids_test.extend(test_data.iloc[:,i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\n\nwifi_bssids_size = len(wifi_bssids_test)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids.extend(wifi_bssids_test)\nwifi_bssids_size = len(wifi_bssids)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"le = LabelEncoder()\nle.fit(wifi_bssids)\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\n\nss = StandardScaler()\nss.fit(data.loc[:,RSSI_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    data.loc[:,i] = le.transform(data.loc[:,i])\n    data.loc[:,i] = data.loc[:,i] + 1\n    \ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\n\ndata.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    test_data.loc[:,i] = le.transform(test_data.loc[:,i])\n    test_data.loc[:,i] = test_data.loc[:,i] + 1\n    \ntest_data.loc[:, 'site_id'] = le_site.transform(test_data.loc[:, 'site_id'])\n\ntest_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"site_count = len(data['site_id'].unique())\ndata.reset_index(drop=True, inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set_seed(SEED)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The model\nThe first Embedding layer is very important. <br>\nThanks to the layer, we can make sense of these BSSID features. <br>\n<br>\nWe concatenate all the features and put them into LSTM. <br>\n<br>\nIf something is theoritically wrong, please correct me. Thank you in advance. ","metadata":{}},{"cell_type":"code","source":"def create_model(input_data):\n    # bssid feats\n    input_dim = input_data[0].shape[1]\n\n    input_embd_layer = L.Input(shape=(input_dim,))\n    x1 = L.Embedding(wifi_bssids_size, 64)(input_embd_layer)\n    x1 = L.Flatten()(x1)\n    # rssi feats\n    input_dim = input_data[1].shape[1]\n\n    input_layer = L.Input(input_dim, )\n    \n    x2 = L.BatchNormalization()(input_layer)\n    x2 = L.Dense(NUM_FEATS * 64, activation='selu')(x2)\n    x2 = L.LeakyReLU()(x2)\n\n    # site\n    input_site_layer = L.Input(shape=(1,))\n    x3 = L.Embedding(site_count, 8)(input_site_layer)\n    x3 = L.Flatten()(x3)\n\n    x = L.Concatenate(axis=1)([x1, x3, x2])\n\n    x = L.BatchNormalization()(x)\n    x = L.Dropout(0.3)(x)\n    x = WeightNormalization(L.Dense(512,activation='selu'))(x)\n    \n    x = L.BatchNormalization()(x)\n    x = L.Dropout(0.3)(x)\n    x = WeightNormalization(L.Dense(256,activation='selu'))(x)\n    \n    x = L.BatchNormalization()(x)\n    x = L.Dropout(0.3)(x)\n    x = WeightNormalization(L.Dense(128, activation='selu'))(x)\n    \n    output_layer_1 = L.Dense(2, name='xy')(x)\n    output_layer_2 = L.Dense(1, activation='softmax', name='floor')(x)\n\n    model = M.Model([input_embd_layer, input_layer, input_site_layer], \n                    [output_layer_1, output_layer_2])\n\n    model.compile(optimizer=tf.optimizers.Adam(lr=0.001),\n                  loss='mse', metrics=['mse'])\n\n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_df = pd.DataFrame()\noof = list()\npredictions = list()\n\noof_x, oof_y, oof_f = np.zeros(data.shape[0]), np.zeros(data.shape[0]), np.zeros(data.shape[0])\npreds_x, preds_y = 0, 0\npreds_f_arr = np.zeros((test_data.shape[0], N_SPLITS))\n\nfor fold, (trn_idx, val_idx) in enumerate(StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED).split(data.loc[:, 'path'], data.loc[:, 'path'])):\n    X_train = data.loc[trn_idx, BSSID_FEATS + RSSI_FEATS + ['site_id']]\n    y_trainx = data.loc[trn_idx, 'x']\n    y_trainy = data.loc[trn_idx, 'y']\n    y_trainf = data.loc[trn_idx, 'floor']\n\n    tmp = pd.concat([y_trainx, y_trainy], axis=1)\n    y_train = [tmp, y_trainf]\n\n    X_valid = data.loc[val_idx, BSSID_FEATS + RSSI_FEATS + ['site_id']]\n    y_validx = data.loc[val_idx, 'x']\n    y_validy = data.loc[val_idx, 'y']\n    y_validf = data.loc[val_idx, 'floor']\n\n    tmp = pd.concat([y_validx, y_validy], axis=1)\n    y_valid = [tmp, y_validf]\n\n    model = create_model([X_train.loc[:,BSSID_FEATS], X_train.loc[:,RSSI_FEATS], X_train.loc[:,'site_id']])\n    model.fit([X_train.loc[:,BSSID_FEATS], X_train.loc[:,RSSI_FEATS], X_train.loc[:,'site_id']], y_train, \n                validation_data=([X_valid.loc[:,BSSID_FEATS], X_valid.loc[:,RSSI_FEATS], X_valid.loc[:,'site_id']], y_valid), \n                batch_size=batch_size, epochs=epochs,\n                callbacks=[\n                  ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, verbose=1, min_delta=1e-4, mode='min')\n                , ModelCheckpoint(f'{base_path}/MLP_{SEED}_{fold}.hdf5', monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='min')\n                , EarlyStopping(monitor='val_loss', min_delta=1e-4, patience=10, mode='min', baseline=None, restore_best_weights=True)\n            ])\n    \n    model.load_weights(f'{base_path}/MLP_{SEED}_{fold}.hdf5')\n    val_pred = model.predict([X_valid.loc[:,BSSID_FEATS], X_valid.loc[:,RSSI_FEATS], X_valid.loc[:,'site_id']])\n\n    oof_x[val_idx] = val_pred[0][:,0]\n    oof_y[val_idx] = val_pred[0][:,1]\n    oof_f[val_idx] = val_pred[1][:,0].astype(int)\n\n    pred = model.predict([test_data.loc[:,BSSID_FEATS], test_data.loc[:,RSSI_FEATS], test_data.loc[:,'site_id']]) # test_data.iloc[:, :-1])\n    preds_x += pred[0][:,0]\n    preds_y += pred[0][:,1]\n    preds_f_arr[:, fold] = pred[1][:,0].astype(int)\n\n    score = comp_metric(oof_x[val_idx], oof_y[val_idx], oof_f[val_idx],\n                        y_validx.to_numpy(), y_validy.to_numpy(), y_validf.to_numpy())\n    oof.append(score)\n    print(f\"fold {fold}: mean position error {score}\")\n\n\npreds_x /= (fold + 1)\npreds_y /= (fold + 1)\n    \nprint(\"*+\"*40)\nprint(f\"mean position error {sum(oof)/N_SPLITS}\")\nprint(\"*+\"*40)\n\npreds_f_mode = stats.mode(preds_f_arr, axis=1)\npreds_f = preds_f_mode[0].astype(int).reshape(-1)\ntest_preds = pd.DataFrame(np.stack((preds_f, preds_x, preds_y))).T\ntest_preds.columns = subm.columns\ntest_preds.index = test_data[\"site_path_timestamp\"]\ntest_preds[\"floor\"] = test_preds[\"floor\"].astype(int)\npredictions.append(test_preds)\n\nall_preds = pd.concat(predictions)\nall_preds = all_preds.reindex(subm.index)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"mean position error {sum(oof)/N_SPLITS}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fix the floor prediction\nSo far, it is not successfully make the \"floor\" prediction part with this dataset. <br>\nTo make it right, we can incorporate [@nigelhenry](https://www.kaggle.com/nigelhenry/)'s [excellent work](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model). <br>","metadata":{}},{"cell_type":"code","source":"all_preds.to_csv('submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"simple_accurate_99 = pd.read_csv('../input/simple-99-accurate-floor-model/submission.csv')\nall_preds['floor'] = simple_accurate_99['floor'].values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"simple_accurate_99.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_preds.to_csv('simple_accurate_99_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's it. \n\nThank you for reading all of it.\n\nI hope it helps!\n\nPlease make comments if you found something to point out, insights or suggestions. ","metadata":{}}]}