{"cells":[{"metadata":{},"cell_type":"markdown","source":"This notebook is a slight modification of \n[LSTM by Keras with Unified Wi-Fi Feats](https://www.kaggle.com/kokitanisaka/lstm-by-keras-with-unified-wi-fi-feats/execution) by [@Kouki](https://www.kaggle.com/kokitanisaka) and [LSTM, Unified Wi-Fi, training x and y with floor](https://www.kaggle.com/therocket290/lstm-unified-wi-fi-training-x-and-y-with-floor) by [@therocket290](https://www.kaggle.com/therocket290)<br> \n\nFloor predictions are pretty accurate, so I wanted to test training the $x$ and $y$ coordinates with the floor data. The only thing I changed was training the $x$ and $y$ models with the floor data included. I add the floor predictions from [Simple 👌 99% Accurate Floor Model 💯](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model) to the test data before making the $x$ and $y$ predictions."},{"metadata":{},"cell_type":"markdown","source":"## Overview\n\nIt demonstrates how to utilize [the unified Wi-Fi dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\nThe Neural Net model is not optimized, there's much space to improve the score. \n\nIn this notebook, I refer these two excellent notebooks.\n* [wifi features with lightgbm/KFold](https://www.kaggle.com/hiro5299834/wifi-features-with-lightgbm-kfold) by [@hiro5299834](https://www.kaggle.com/hiro5299834/)<br>\n I took some code fragments from his notebook.\n* [Simple 👌 99% Accurate Floor Model 💯](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model) by [@nigelhenry](https://www.kaggle.com/nigelhenry/)<br>\n I use his excellent work, the \"floor\" prediction.\n\nThank you!"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\n\nimport random\nimport os\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nimport tensorflow.keras.backend as K\nimport tensorflow_addons as tfa\nfrom tensorflow_addons.layers import WeightNormalization\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping, TensorBoard\nimport datetime","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### options\nWe can change the way it learns with these options. <br>\nEspecialy **NUM_FEATS** is one of the most important options. <br>\nIt determines how many features are used in the training. <br>\nWe have 100 Wi-Fi features in the dataset, but 100th Wi-Fi signal sounds not important, right? <br>\nSo we can use top Wi-Fi signals if we think we need to. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# options\n\nN_SPLITS = 10\n\nSEED = 2021\n\nNUM_FEATS = 20 # number of features that we use. there are 100 feats but we don't need to use all of them\n\nbase_path = '/kaggle'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n    \ndef comp_metric(xhat, yhat, fhat, x, y, f):\n    intermediate = np.sqrt(np.power(xhat-x, 2) + np.power(yhat-y, 2)) + 15 * np.abs(fhat-f)\n    return intermediate.sum()/xhat.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_dir = f\"{base_path}/input/indoorunifiedwifids\"\ntrain_files = sorted(glob.glob(os.path.join(feature_dir, '*_train.csv')))\ntest_files = sorted(glob.glob(os.path.join(feature_dir, '*_test.csv')))\nsubm = pd.read_csv(f'{base_path}/input/indoor-location-navigation/sample_submission.csv', index_col=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(f'{feature_dir}/train_all.pkl', 'rb') as f:\n  data = pickle.load( f)\n\nwith open(f'{feature_dir}/test_all.pkl', 'rb') as f:\n  test_data = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# training target features\n\nBSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS  = [f'rssi_{i}' for i in range(NUM_FEATS)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get numbers of bssids to embed them in a layer\n\nwifi_bssids = []\nfor i in range(100):\n    wifi_bssids.extend(data.iloc[:,i].values.tolist())\nwifi_bssids = list(set(wifi_bssids))\n\nwifi_bssids_size = len(wifi_bssids)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids_test = []\nfor i in range(100):\n    wifi_bssids_test.extend(test_data.iloc[:,i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\n\nwifi_bssids_size = len(wifi_bssids_test)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids.extend(wifi_bssids_test)\nwifi_bssids_size = len(wifi_bssids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# preprocess\n\nle = LabelEncoder()\nle.fit(wifi_bssids)\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\n\nss = StandardScaler()\nss.fit(data.loc[:,RSSI_FEATS+['floor']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.loc[:,RSSI_FEATS+['floor']] = ss.transform(data.loc[:,RSSI_FEATS+['floor']])\nfor i in BSSID_FEATS:\n    data.loc[:,i] = le.transform(data.loc[:,i])\n    data.loc[:,i] = data.loc[:,i] + 1\n    \ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\n\ndata.loc[:,RSSI_FEATS+['floor']] = ss.transform(data.loc[:,RSSI_FEATS+['floor']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add floor predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"simple_accurate_99 = pd.read_csv('../input/simple-99-accurate-floor-model/submission.csv') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data['floor'] = simple_accurate_99['floor'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.loc[:,RSSI_FEATS+['floor']] = ss.transform(test_data.loc[:,RSSI_FEATS+['floor']])\nfor i in BSSID_FEATS:\n    test_data.loc[:,i] = le.transform(test_data.loc[:,i])\n    test_data.loc[:,i] = test_data.loc[:,i] + 1\n    \ntest_data.loc[:, 'site_id'] = le_site.transform(test_data.loc[:, 'site_id'])\n\ntest_data.loc[:,RSSI_FEATS+['floor']] = ss.transform(test_data.loc[:,RSSI_FEATS+['floor']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site_count = len(data['site_id'].unique())\ndata.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The model\nThe first Embedding layer is very important. <br>\nThanks to the layer, we can make sense of these BSSID features. <br>\n<br>\nWe concatenate all the features and put them into GRU. <br>\n<br>\nIf something is theoritically wrong, please correct me. Thank you in advance. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model(input_data):\n\n    # bssid feats\n    input_dim = input_data[0].shape[1]\n\n    input_embd_layer = L.Input(shape=(input_dim,), name='Input_bssid')\n    x1 = L.Embedding(wifi_bssids_size, 64 , name='Embedding_bssid')(input_embd_layer)\n    x1 = L.Flatten(name='Flatten_bssid')(x1)\n\n    # rssi feats\n    input_dim = input_data[1].shape[1]\n\n    input_layer = L.Input(input_dim, name='Input_rssi')\n    x2 = L.BatchNormalization(name='BatchNormalization_rssi')(input_layer)\n    x2 = L.Dense(NUM_FEATS * 64, activation='relu' , name='Dense_rssi')(x2)\n    ### self add\n    #x2 = L.Dense(NUM_FEATS * 64, name='Dense_rssi')(x2)\n    #x2 = L.LeakyReLU(alpha=0.2, name='LeakyReLU_rssi')(x2)\n    \n    # site\n    input_site_layer = L.Input(shape=(1,), name='Input_site')\n    x3 = L.Embedding(site_count, 1, name='Embedding_site')(input_site_layer)\n    x3 = L.Flatten(name='Flatten_site')(x3)\n\n    # main stream\n    x = L.Concatenate(axis=1, name='Concatenate_allinput')([x1, x3, x2])\n\n    x = L.BatchNormalization(name='BatchNormalization_main_1')(x)\n    x = L.Dropout(0.25, name='Dropout_main')(x)\n    #x = L.Dense(256, activation='relu')(x)\n\n    ### self add\n    x = L.Dense(256, name='Dense_main')(x)\n    x = L.LeakyReLU(alpha=0.1, name='LeakyReLU_main')(x)\n    \n    x = L.Reshape((1, -1), name='Reshape_main')(x)\n    x = L.BatchNormalization(name='BatchNormalization_main_2')(x)\n    #x = L.LSTM(128, dropout=0.3, recurrent_dropout=0.3, return_sequences=True, activation='relu')(x)\n    x = L.GRU(256, dropout=0.3, recurrent_dropout=0.3, return_sequences=True, activation='relu', name='GRU_main_1')(x)\n    x = L.GRU(16, dropout=0.1, return_sequences=False, activation='relu', name='GRU_main_2')(x)\n    \n    output_layer_1 = L.Dense(2 ,name='Output_xy')(x)\n    #output_layer_2 = L.Dense(1, activation='softmax', name='floor')(x)\n\n    model = M.Model([input_embd_layer, input_layer, input_site_layer], \n                    [output_layer_1])\n\n    model.compile(optimizer=tf.optimizers.RMSprop(lr=0.001),\n                  loss='mse', metrics=['mse'])\n    model.summary()\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%load_ext tensorboard\n\n!rm -rf ./logs/ \n!mkdir ./logs/\n\n!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n!unzip ngrok-stable-linux-amd64.zip\n\n# Run tensorboard as well as Ngrox (for tunneling as non-blocking processes)\nimport multiprocessing\n\n\npool = multiprocessing.Pool(processes = 10)\nresults_of_processes = [pool.apply_async(os.system, args=(cmd, ), callback = None )\n                        for cmd in [\n                        f\"tensorboard --logdir ./logs/ --host 0.0.0.0 --port 6006 &\",\n                        \"./ngrok http 6006 &\"\n                        ]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! curl -s http://localhost:4040/api/tunnels | python3 -c \\\n    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score_df = pd.DataFrame()\npredictions = list()\n\npreds_x, preds_y = 0, 0\npreds_f_arr = np.zeros((test_data.shape[0], N_SPLITS))\n\nfor fold, (trn_idx, val_idx) in enumerate(StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED).split(data.loc[:, 'path'], data.loc[:, 'path'])):\n    X_train = data.loc[trn_idx, BSSID_FEATS + RSSI_FEATS + ['floor','site_id']]\n    y_trainx = data.loc[trn_idx, 'x']\n    y_trainy = data.loc[trn_idx, 'y']\n    y_trainf = data.loc[trn_idx, 'floor']\n\n    tmp = pd.concat([y_trainx, y_trainy], axis=1)\n    #y_train = [tmp, y_trainf]\n    y_train = tmp\n\n    X_valid = data.loc[val_idx, BSSID_FEATS + RSSI_FEATS + ['floor','site_id']]\n    y_validx = data.loc[val_idx, 'x']\n    y_validy = data.loc[val_idx, 'y']\n    y_validf = data.loc[val_idx, 'floor']\n\n    tmp = pd.concat([y_validx, y_validy], axis=1)\n    #y_valid = [tmp, y_validf]\n    y_valid = tmp\n\n    model = create_model([X_train.loc[:,BSSID_FEATS], X_train.loc[:,RSSI_FEATS+['floor']], X_train.loc[:,'site_id']])\n    \n    log_dir = \"logs/fit/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n    tensorboard_cb = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n    \n    history = model.fit([X_train.loc[:,BSSID_FEATS], X_train.loc[:,RSSI_FEATS+['floor']], X_train.loc[:,'site_id']], y_train, \n                validation_data=([X_valid.loc[:,BSSID_FEATS], X_valid.loc[:,RSSI_FEATS+['floor']], X_valid.loc[:,'site_id']], y_valid), \n                batch_size=128, epochs=1000,\n                callbacks=[\n                ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=5, verbose=1, min_delta=1e-4, mode='min')\n                , ModelCheckpoint(f'{base_path}/working/RNN_{SEED}_{fold}.hdf5', monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='min')\n                , EarlyStopping(monitor='val_loss', min_delta=1e-4, patience=8, mode='min', baseline=None, restore_best_weights=True)\n                , tensorboard_cb\n            ])\n\n    model.load_weights(f'{base_path}/working/RNN_{SEED}_{fold}.hdf5')\n    #val_pred = model.predict([X_valid.loc[:,BSSID_FEATS], X_valid.loc[:,RSSI_FEATS], X_valid.loc[:,'site_id'], X_valid.loc[:,'floor']])\n\n    pred = model.predict([test_data.loc[:,BSSID_FEATS], test_data.loc[:,RSSI_FEATS+['floor']], test_data.loc[:,'site_id']]) # test_data.iloc[:, :-1])\n    preds_x += pred[:,0]\n    preds_y += pred[:,1]\n    #preds_f_arr[:, fold] = pred[1][:,0].astype(int)\n\n    \n\n    break # for demonstration, run just one fold as it takes much time.\n\npreds_x /= (fold + 1)\npreds_y /= (fold + 1)\n    \nprint(\"*+\"*40)\nprint(\"*+\"*40)\n\n#preds_f_mode = stats.mode(preds_f_arr, axis=1)\n#preds_f = preds_f_mode[0].astype(int).reshape(-1)\npreds_f = test_data['floor']\ntest_preds = pd.DataFrame(np.stack((preds_f, preds_x, preds_y))).T\ntest_preds.columns = subm.columns\ntest_preds.index = test_data[\"site_path_timestamp\"]\ntest_preds[\"floor\"] = test_preds[\"floor\"].astype(int)\npredictions.append(test_preds)\n\n# %tensorboard --logdir logs/fit","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_preds = pd.concat(predictions)\nall_preds = all_preds.reindex(subm.index)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fix the floor prediction\nSo far, it is not successfully make the \"floor\" prediction part with this dataset. <br>\nTo make it right, we can incorporate [@nigelhenry](https://www.kaggle.com/nigelhenry/)'s [excellent work](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model). <br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"simple_accurate_99 = pd.read_csv('../input/simple-99-accurate-floor-model/submission.csv')\n\nall_preds['floor'] = simple_accurate_99['floor'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_preds.to_csv('/kaggle/working/submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_train_history(train_history,train,validation):\n    plt.plot(train_history.history[train])\n    plt.plot(train_history.history[validation])\n    plt.title('Train History')\n    plt.ylabel(train)\n    plt.xlabel(\"Epoch\")\n    plt.legend(['train '+train,'validation '+train],loc='upper right')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nprint(history.history.keys())\nshow_train_history(history, 'loss', 'val_loss')\nshow_train_history(history, 'mse', 'val_mse')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(history.history['lr'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('Train History')\nplt.ylabel('loss')\nplt.xlabel(\"Epoch\")\nplt.ylim([0,150])  # y軸邊界\nplt.legend(['train '+'loss','validation '+'loss'],loc='upper right')\nplt.show()\nplt.plot(history.history['mse'])\nplt.plot(history.history['val_mse'])\nplt.title('Train History')\nplt.ylabel('mse')\nplt.xlabel(\"Epoch\")\nplt.ylim([0,150])  # y軸邊界\nplt.legend(['train '+'mse','validation '+'mse'],loc='upper right')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's it. \n\nThank you for reading all of it.\n\nPlease make comments if you found something to point out, insights or suggestions. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}