{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## The final form of my LSTM\n## Overview\n2 months ago, I published [this notebook](https://www.kaggle.com/kokitanisaka/lstm-by-keras-with-unified-wi-fi-feats), predicting waypoints by LSTM.\n\nAfter that, I kept working on with the model and this notebook is the final form of the notebook.\n\nI incorporated self-attention in it and the result is better than the last one.\n\nActually this one doesn't perform well like other solutions, but if you are familiar with Keras and don't know how to apply self-attention in your model, it can help you.\n\n## How does the model look like?\nIt looks like ths way.\n\n! Some details are omitted. Too see the details, please take a look at the code. \n\n<img src= \"https://i.imgur.com/bH76DpW.png\" alt =\"the structure of the model\" style='width: 500px;'>\n\n* delta: This feature was extracted by host's function and was used in [Saito's notebook](https://www.kaggle.com/saitodevel01/indoor-post-processing-by-cost-minimization). \n* user id: This feature was found by [tomoo](https://www.kaggle.com/tomooinubushi/retrieving-user-id-from-leaked-wifi-feature).\n* time gap: It came from Wi-Fi observations. This is calculated by subtracting **timestamp** from **last seen timestamp**. As this feature can't be calculated for the test set, we needed to retrieve the original **timestamp** for test set. Timestamp was fully extracted by the team mate, [Housuke](https://www.kaggle.com/horsek).\n\n## Some notes about the model\n\n* Thanks to self-attention, I succeeded to use all the 100 features. Before introfucing self-attention, when I put more features than 20, the result got worse. It seems LSTM can't handle that much features in this case.\n* It predicts **floor** but the accuracy is awful. The reason why I keep predicting floor with model is, it helps predicting **x** and **y**. And feeding **floor** as a feature didn't work for me.\n* For **BSSID** feature, I introduced mask in it. If we take a look at the dataset, we can see **-999** in **RSSI** features. It means no signals are observed. In this case, these **BSSIDs** shouldn't be learned by the model. \n* This notebook takes much time to finish. One epoch takes around 500 sec and epochs are around 120 for each fold. So it won't finish in 9 hours. To tackle this issue, I introduced some functions in this notebook.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\n\nimport psutil\nimport random\nimport os\nimport time\nimport sys\nimport math\nfrom contextlib import contextmanager\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.decomposition import PCA\n\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nimport tensorflow.keras.backend as K\nimport tensorflow_addons as tfa\nfrom tensorflow_addons.layers import WeightNormalization\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Options\n* TRIAL_ROUND : The number of trials. If it was second, put 1 here.\n* PLATEAU : The number of **ReduceLROnPlateau** happened in the last trial. \n* TARGET_FOLDS : The fold which tackle on this training. As only one fold won't finish in 9 hours, we can put number of folds and run each folds in different notebooks at the same time.","metadata":{}},{"cell_type":"code","source":"MODEL_NAME='TF_comp_user_fix'\n\nSEED = 47\nN_SPLITS = 10\n\nNUM_FEATS = 100\nN_COMPONENTS = 15\n\nINFERENCE_MODE = True # turn it into False if we want to try training\nMODEL_DATASET='indoor-models' # for the training after 2 rounds, former models must be loaded from this dataset\n\nTARGET_FOLDS = [0]\nMAX_EPOCHS = 36\nPLATEAU = 0\nTRIAL_ROUND = 0","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# utils\n@contextmanager\ndef timer(name: str):\n    t0 = time.time()\n    p = psutil.Process(os.getpid())\n    m0 = p.memory_info()[0] / 2. ** 30\n    try:\n        yield\n    finally:\n        m1 = p.memory_info()[0] / 2. ** 30\n        delta = m1 - m0\n        sign = '+' if delta >= 0 else '-'\n        delta = math.fabs(delta)\n        print(f\"[{m1:.1f}GB({sign}{delta:.1f}GB): {time.time() - t0:.3f}sec] {name}\", file=sys.stderr)\n\n\ndef set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)","metadata":{"_cell_guid":"c9209aed-2e70-43b3-9212-e880a98972e0","_uuid":"c2b678b0-6a72-4962-8ca9-bedf0bd9c953","papermill":{"duration":8.618146,"end_time":"2021-04-27T08:02:42.242551","exception":false,"start_time":"2021-04-27T08:02:33.624405","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set_seed(SEED)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Preparation","metadata":{}},{"cell_type":"code","source":"with open(f'../input/indoor-with-delta/train_all.pkl', 'rb') as f:\n    data = pickle.load(f)\nwith open(f'../input/indoor-interpolated-with-gap/test_all.pkl', 'rb') as f:\n    test_data = pickle.load(f)","metadata":{"_cell_guid":"8ea86bc6-4fe8-4f93-89f7-2bafd0dca332","_uuid":"7d82a9e0-e874-42ec-a408-f38d0cbab7ab","papermill":{"duration":23.371487,"end_time":"2021-04-27T08:03:05.946131","exception":false,"start_time":"2021-04-27T08:02:42.574644","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS  = [f'rssi_{i}' for i in range(NUM_FEATS)]\nGAP_FEATS  = [f'gap_{i}' for i in range(NUM_FEATS)]\nDELTA_FEATS = ['delta_x_hat', 'delta_y_hat']","metadata":{"papermill":{"duration":0.030474,"end_time":"2021-04-27T08:02:42.390752","exception":false,"start_time":"2021-04-27T08:02:42.360278","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data['site_id'] = data['site']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wifi_bssids = []\nfor i in BSSID_FEATS:\n    wifi_bssids.extend(data.loc[:,i].values.tolist())\nwifi_bssids = list(set(wifi_bssids))\n\nwifi_bssids_size = len(wifi_bssids)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids_test = []\nfor i in BSSID_FEATS:\n    wifi_bssids_test.extend(test_data.loc[:,i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\n\nwifi_bssids_size = len(wifi_bssids_test)\nprint(f'BSSID TYPES: {wifi_bssids_size}')","metadata":{"_cell_guid":"d0222a09-978d-4aea-9c11-be3774a8d065","_uuid":"fdb3083b-a389-4968-b84e-e96fee2fc71e","papermill":{"duration":4.555073,"end_time":"2021-04-27T08:03:10.633966","exception":false,"start_time":"2021-04-27T08:03:06.078893","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wifi_bssids.extend(wifi_bssids_test)\nwifi_bssids = list(set(wifi_bssids))\nwifi_bssids_size = len(wifi_bssids)","metadata":{"papermill":{"duration":0.063146,"end_time":"2021-04-27T08:03:10.720755","exception":false,"start_time":"2021-04-27T08:03:10.657609","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"le = LabelEncoder()\nle.fit(wifi_bssids)\n\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\n\ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\nfor i in BSSID_FEATS:\n    data.loc[:,i] = le.transform(data.loc[:,i])\n    data.loc[:,i] = data.loc[:,i] + 1\n\ntest_data.loc[:, 'site_id'] = le_site.transform(test_data.loc[:, 'site_id'])\nfor i in BSSID_FEATS:\n    test_data.loc[:,i] = le.transform(test_data.loc[:,i])\n    test_data.loc[:,i] = test_data.loc[:,i] + 1\n\nsite_count = len(data['site_id'].unique())","metadata":{"_cell_guid":"e7293fdd-fee0-417f-86ea-a7194546cc72","_uuid":"5f899bcf-e1cd-4450-8b91-d3ce61bd96b0","papermill":{"duration":173.462844,"end_time":"2021-04-27T08:06:04.269058","exception":false,"start_time":"2021-04-27T08:03:10.806214","status":"completed"},"scrolled":true,"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Delete some records that the distance is too big. \n\nDistance is calculated using delta x and delta y.","metadata":{}},{"cell_type":"code","source":"sort = data.sort_values(['path', 'timestamp'])\nsort['x_shift'] = sort.groupby(['path'])['x'].shift()\nsort['y_shift'] = sort.groupby(['path'])['y'].shift()\nsort['dist'] = sort.apply(lambda x: math.sqrt(x['delta_x_hat'] ** 2 + x['delta_y_hat'] ** 2 ), axis = 1)\nsort = sort.sort_index()\n\nsort = sort[sort['dist'] >= 25][['x', 'x_shift', 'delta_x_hat', 'y', 'y_shift', 'delta_y_hat', 'dist']]\nsort['delta_x_hat'] = sort.apply(lambda x: x['x'] - x['x_shift'], axis=1)\nsort['delta_y_hat'] = sort.apply(lambda x: x['y'] - x['y_shift'], axis=1)\n\ndata.loc[sort.index, 'delta_x_hat'] = sort['delta_x_hat'].values\ndata.loc[sort.index, 'delta_y_hat'] = sort['delta_y_hat'].values","metadata":{"papermill":{"duration":11.891966,"end_time":"2021-04-27T08:06:16.185475","exception":false,"start_time":"2021-04-27T08:06:04.293509","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ss = StandardScaler()\nss.fit(data[DELTA_FEATS])\nss.transform(data[DELTA_FEATS])\ndata[DELTA_FEATS] = ss.transform(data[DELTA_FEATS])","metadata":{"papermill":{"duration":0.192671,"end_time":"2021-04-27T08:06:16.402489","exception":false,"start_time":"2021-04-27T08:06:16.209818","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Mask the useless BSSIDs. \n\nIf a RSSI value was -999, it means the Wi-Fi signal wasn't observed in the waypoint.\n\nWe don't want the NN to learn these meaningless BSSIDs so we mask them.","metadata":{}},{"cell_type":"code","source":"a = data[BSSID_FEATS]\na.columns = [str(i) for i in range(len(BSSID_FEATS))]\n\nb = data[RSSI_FEATS]\nb.columns = [str(i) for i in range(len(BSSID_FEATS))]\n\nx = a.mask(b == -999, 0)\nx.columns = BSSID_FEATS\ndata[BSSID_FEATS] = x\n\na = test_data[BSSID_FEATS]\na.columns = [str(i) for i in range(len(BSSID_FEATS))]\n\nb = test_data[RSSI_FEATS]\nb.columns = [str(i) for i in range(len(BSSID_FEATS))]\n\nx = a.mask(b == -999, 0)\nx.columns = BSSID_FEATS\ntest_data[BSSID_FEATS] = x","metadata":{"_cell_guid":"165ccd4c-7f90-436e-b9aa-5e0a4e980273","_uuid":"d876eb76-8ab3-4e2b-b581-557c002716b6","papermill":{"duration":4.984718,"end_time":"2021-04-27T08:06:21.412941","exception":false,"start_time":"2021-04-27T08:06:16.428223","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_rssis = data['rssi_0']\nfor i in RSSI_FEATS[1:]:\n    all_rssis = pd.concat([all_rssis, data[i]])\n\nss = StandardScaler()\nss.fit(pd.DataFrame(all_rssis))\n\nfor i in RSSI_FEATS:\n    data.loc[:,i] = ss.transform(pd.DataFrame(data.loc[:,i]))\n    test_data.loc[:,i] = ss.transform(pd.DataFrame(test_data.loc[:,i]))\n    \nall_rssis = data['gap_0']\nfor i in GAP_FEATS[1:]:\n    all_rssis = pd.concat([all_rssis, data[i]])\n\nss = StandardScaler()\nss.fit(pd.DataFrame(all_rssis))\n\nfor i in GAP_FEATS:\n    data.loc[:,i] = ss.transform(pd.DataFrame(data.loc[:,i]))\n    test_data.loc[:,i] = ss.transform(pd.DataFrame(test_data.loc[:,i]))","metadata":{"_cell_guid":"ae8633cd-da11-482e-8c60-9b233e2658a8","_uuid":"211a3636-8648-4d8f-bc90-a069c98d1056","papermill":{"duration":196.474712,"end_time":"2021-04-27T08:09:37.913875","exception":false,"start_time":"2021-04-27T08:06:21.439163","status":"completed"},"scrolled":true,"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Yield PCA features from RSSI features.","metadata":{}},{"cell_type":"code","source":"PCA_COLUMNS = [f'rssi_pca_{i}' for i in range(N_COMPONENTS)]\n\npca = PCA(n_components=N_COMPONENTS, random_state=SEED)\npca.fit(data.loc[:,RSSI_FEATS])\n\ndata_pca = pd.DataFrame(pca.transform(data.loc[:,RSSI_FEATS]))\ndata_pca.columns = PCA_COLUMNS\ndata = pd.concat([data, data_pca], axis=1)\n\ntest_pca = pd.DataFrame(pca.transform(test_data.loc[:,RSSI_FEATS]))\ntest_pca.columns = PCA_COLUMNS\ntest_data = pd.concat([test_data, test_pca], axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"floor_count = len(data['floor'].unique())\ndata['floor'] = data['floor'].astype(int)\ny = pd.get_dummies(data.loc[:,'floor'])","metadata":{"papermill":{"duration":0.049886,"end_time":"2021-04-27T08:09:37.992325","exception":false,"start_time":"2021-04-27T08:09:37.942439","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most user_ids are observed only in train set, I masked user_id which is not observed in test set. ","metadata":{}},{"cell_type":"code","source":"user_id = pd.read_csv('../input/retrieving-user-id-from-leaked-wifi-feature/df.csv')\ndata = data.merge(user_id[['path_id', 'user_id']], left_on='path', right_on='path_id', how='left')\n\ntest_data['path'] = test_data['site_path_timestamp'].apply(lambda x: x.split('_')[1])\ntest_data = test_data.merge(user_id[['path_id', 'user_id']], left_on='path', right_on='path_id', how='left')\n\ndata['user_id'] = data['user_id'] + 1\ntest_data['user_id'] = test_data['user_id'] + 1\n\nshared_user_ids = (set(data['user_id'].unique()) & set(test_data['user_id']))\nprint(len(shared_user_ids))\n\ndata['user_id'] = data['user_id'].apply(lambda x: x if x in shared_user_ids else 0)\ntest_data['user_id'] = test_data['user_id'].apply(lambda x: x if x in shared_user_ids else 0)\n\nkey_map = {j: i for (i, j) in enumerate(data['user_id'].unique())}\n\ndata['user_id'] = data['user_id'].apply(lambda x: key_map[x])\ntest_data['user_id'] = test_data['user_id'].apply(lambda x: key_map[x])\n\nuserid_count = len(shared_user_ids)","metadata":{"papermill":{"duration":0.916726,"end_time":"2021-04-27T08:09:39.361431","exception":false,"start_time":"2021-04-27T08:09:38.444705","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Set delta features for test set.\n\ndelta features are made based on other prediction. ","metadata":{}},{"cell_type":"code","source":"test_delta = pd.read_csv('../input/indoor-with-delta/delta_for_test_from_4.006.csv')\ntest_data = test_data.merge(test_delta, on='site_path_timestamp', how='left')\n\nss_delta = StandardScaler()\nss_delta.fit(data[DELTA_FEATS])\nss_delta.transform(data[DELTA_FEATS])\ndata[DELTA_FEATS] = ss_delta.transform(data[DELTA_FEATS])\ntest_data[DELTA_FEATS] = ss_delta.transform(test_data[DELTA_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"def create_model(input_data):\n\n    # bssid feats\n    input_dim = input_data[0].shape[1]\n\n    input_embd_layer = L.Input(shape=(input_dim,))\n    x1 = L.Embedding(wifi_bssids_size + 1,128, mask_zero=True)(input_embd_layer)\n    x1 = L.Flatten()(x1)\n\n    # site\n    input_site_layer = L.Input(shape=(1,))\n    x3 = L.Embedding(site_count, 2)(input_site_layer)\n    x3 = L.Flatten()(x3)\n\n    # rssi feats\n    input_dim = input_data[2].shape[1]\n\n    input_layer_2 = L.Input(input_dim, )\n    x4 = L.BatchNormalization()(input_layer_2)\n    x4 = L.Dense(32, activation='swish')(x4)\n\n    # delta feats\n    input_dim = input_data[3].shape[1]\n\n    input_layer_3 = L.Input(input_dim, )\n    x6 = L.BatchNormalization()(input_layer_3)\n    x6 = L.Dense(256, activation='swish')(x6)\n    x6 = L.Reshape((1, 256))(x6)\n    \n    \n    # user_id\n    input_userid_layer = L.Input(shape=(1,))\n    x7 = L.Embedding(userid_count + 1, 4, mask_zero=True)(input_userid_layer)\n    x7 = L.Flatten()(x7)    \n    \n    input_rssi_gap = []\n    x5 = []\n    for c in RSSI_FEATS:\n        _i = L.Input(2, )\n        \n        _x5 = L.BatchNormalization()(_i)\n        _x5 = L.Dense(1, activation='swish')(_x5)\n\n        input_rssi_gap.append(_i)\n        x5.append(_x5)\n\n    concatenated = L.Concatenate(axis=1)([x1, x3, x4, x7] + x5)\n    concatenated = L.BatchNormalization()(concatenated)\n    concatenated = L.Dropout(0.4)(concatenated)\n    concatenated = L.Dense(256, activation='swish')(concatenated)\n    concatenated = L.Reshape((1, -1))(concatenated)\n\n    def attention(query_key, res):\n        l = L.MultiHeadAttention(num_heads=4, key_dim=4, dropout=0.5)(query_key, query_key)\n        l = L.LayerNormalization(epsilon=1e-6)(res + l)\n\n        ffl = L.BatchNormalization()(l)\n        ffl = L.Dropout(0.4)(ffl)\n        ffl = L.Dense(256, activation='relu')(ffl)\n        ffl = L.BatchNormalization()(ffl)\n        ffl = L.Dropout(0.3)(ffl)\n        ffl = L.Dense(64, activation='relu')(ffl)\n        ffl = L.BatchNormalization()(ffl)\n        ffl = L.Dropout(0.5)(ffl)\n        ffl = L.Dense(256, activation='relu')(ffl)\n\n        l = L.LayerNormalization(epsilon=1e-6)(l + ffl)\n        \n        return l\n\n    # self attention\n    x = attention(concatenated, concatenated)\n    x = attention(x, concatenated)    \n    \n    x = L.Concatenate(axis=1)([x, x6])\n    x = L.Reshape((8, -1))(x)\n    x = L.LSTM(64, dropout=0.3, recurrent_dropout=0.3, return_sequences=True, activation='relu')(x)\n    x = L.LSTM(16, dropout=0.1, return_sequences=False, activation='swish')(x)\n\n    output_layer_1 = L.Dense(2, name='xy')(x)\n    output_layer_2 = L.Dense(11, activation='softmax', name='floor')(x)\n\n    model = M.Model([input_embd_layer, input_site_layer, input_layer_2, input_layer_3, input_userid_layer] + input_rssi_gap, \n                    [output_layer_1, output_layer_2])\n\n    lr = 0.001 * (0.1 ** PLATEAU)\n    print(f'lr:{lr}')\n    \n    model.compile(optimizer=tf.optimizers.Adam(lr=lr),\n                  loss='mse', metrics=['mse'])\n\n    return model","metadata":{"_cell_guid":"07eac595-3c40-444d-991f-44067d85f0af","_uuid":"e4739548-01e6-41ba-852f-d87da0dc1e62","papermill":{"duration":0.053916,"end_time":"2021-04-27T08:09:40.328613","exception":false,"start_time":"2021-04-27T08:09:40.274697","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_df = pd.DataFrame()\noof = list()\npredictions = list()\n\noof_x, oof_y, oof_f = np.zeros(data.shape[0]), np.zeros(data.shape[0]), np.zeros(data.shape[0])\npreds_x, preds_y = 0, 0\npreds_f_arr = np.zeros((test_data.shape[0], N_SPLITS))\n\nfor fold, (trn_idx, val_idx) in enumerate(StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED).split(data.loc[:, 'path'], data.loc[:, 'path'])):\n    if (not INFERENCE_MODE) & (fold not in TARGET_FOLDS):\n        continue\n    \n    _X_train = data.loc[trn_idx, :]\n    X_train = [_X_train.loc[:,BSSID_FEATS], _X_train.loc[:,'site_id'], _X_train.loc[:,PCA_COLUMNS], _X_train.loc[:,DELTA_FEATS], _X_train.loc[:,'user_id']]\n    for r, g in zip(RSSI_FEATS, GAP_FEATS):\n        X_train.append(pd.DataFrame(_X_train.loc[:,r].values, _X_train.loc[:,g].values).reset_index(drop=False))\n\n    y_trainx = data.loc[trn_idx, 'x']\n    y_trainy = data.loc[trn_idx, 'y']\n    y_trainf = y.loc[trn_idx, :]\n    tmp = pd.concat([y_trainx, y_trainy], axis=1)\n    y_train = [tmp, y_trainf]\n\n    _X_valid = data.loc[val_idx, :]\n    X_valid = [_X_valid.loc[:,BSSID_FEATS], _X_valid.loc[:,'site_id'], _X_valid.loc[:,PCA_COLUMNS], _X_valid.loc[:,DELTA_FEATS], _X_valid.loc[:,'user_id']]\n    for r, g in zip(RSSI_FEATS, GAP_FEATS):\n        X_valid.append(pd.DataFrame(_X_valid.loc[:,r].values, _X_valid.loc[:,g].values).reset_index(drop=False))\n    \n    y_validx = data.loc[val_idx, 'x']\n    y_validy = data.loc[val_idx, 'y']\n    y_validf = y.loc[val_idx, :]\n    tmp = pd.concat([y_validx, y_validy], axis=1)\n    y_valid = [tmp, y_validf]\n\n    with timer(\"fit\"):\n        model = create_model(X_train)\n        if not INFERENCE_MODE:\n            if TRIAL_ROUND >= 1:\n                model.load_weights(f'../input/{MODEL_DATASET}/{MODEL_NAME}_{SEED}_{fold}.hdf5')\n            model.fit(X_train, y_train, \n                        validation_data=(X_valid, y_valid), \n                        batch_size=64, epochs=MAX_EPOCHS + TRIAL_ROUND*MAX_EPOCHS, initial_epoch=TRIAL_ROUND*MAX_EPOCHS,\n                        callbacks=[\n                        ReduceLROnPlateau(monitor='val_xy_loss', factor=0.1, patience=6, verbose=1, min_delta=1e-4, mode='min')\n                        , ModelCheckpoint(f'{MODEL_NAME}_{SEED}_{fold}.hdf5', monitor = 'val_xy_loss', verbose = 0, save_best_only=True, save_weights_only=True, mode='min')\n                        , ModelCheckpoint(f'{MODEL_NAME}_{SEED}_{fold}_latest.hdf5', monitor = 'val_xy_loss', verbose = 0, save_best_only=False, save_weights_only=True, mode='min')\n                        , EarlyStopping(monitor='val_xy_loss', min_delta=1e-4, patience=10, mode='min', baseline = None, restore_best_weights = True)\n                    ])\n\n    if INFERENCE_MODE:\n        model.load_weights(f'../input/{MODEL_DATASET}/{MODEL_NAME}_{SEED}_{fold}.hdf5')\n        val_pred = model.predict(X_valid)\n\n        oof_x[val_idx] = val_pred[0][:,0]\n        oof_y[val_idx] = val_pred[0][:,1]\n\n        _test_data = [test_data.loc[:,BSSID_FEATS], test_data.loc[:,'site_id'], test_data.loc[:,PCA_COLUMNS], test_data.loc[:,DELTA_FEATS], test_data.loc[:,'user_id']]\n        for r, g in zip(RSSI_FEATS, GAP_FEATS):\n            _test_data.append(pd.DataFrame(test_data.loc[:,r].values, test_data.loc[:,g].values).reset_index(drop=False))\n\n        pred = model.predict(_test_data)\n        preds_x += pred[0][:,0]\n        preds_y += pred[0][:,1]","metadata":{"papermill":{"duration":22516.60488,"end_time":"2021-04-27T14:24:56.965446","exception":false,"start_time":"2021-04-27T08:09:40.360566","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Assess the result (Inference mode only","metadata":{}},{"cell_type":"code","source":"def metrics(output_way, output_floor, way, floor):\n    first_term = np.mean(np.sqrt(np.sum((output_way - way)**2, axis = 1)))\n    second_term = 15 * np.mean(np.abs(output_floor - floor))\n    return first_term, second_term\n\ndef compute_cv_score(oof_):\n    output_way = oof_[['pred_x', 'pred_y']].values\n    output_floor = oof_['pred_floor'].values\n    \n    way = oof_[['true_x', 'true_y']].values\n    floor = oof_['true_floor'].values\n    \n    loss_waypoints, loss_floor = metrics(output_way, output_floor, way, floor)\n    return loss_waypoints, loss_floor","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if INFERENCE_MODE:\n    assess = pd.DataFrame()\n    assess['pred_x'] = oof_x\n    assess['pred_y'] = oof_y\n    assess['pred_floor'] = data['floor'].values\n    assess['true_x'] = data['x'].values\n    assess['true_y'] = data['y'].values\n    assess['true_floor'] = data['floor'].values\n    cv = compute_cv_score(assess)\n    print(cv)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if INFERENCE_MODE:\n    preds_x /= N_SPLITS\n    preds_y /= N_SPLITS\n\n    sub = pd.read_csv('../input/indoor-location-navigation/sample_submission.csv')\n    \n    sub['x'] = preds_x\n    sub['y'] = preds_y\n    # floor prediction was made by the other notebook.\n    del sub['floor']\n    sub['path'] = sub['site_path_timestamp'].apply(lambda x: x.split('_')[1])\n    floor = pd.read_csv('../input/indoor-floor-prediction/floor_pred_0507.csv').reset_index(drop=True)[['path', 'floor']]\n    sub = sub.merge(floor, on=['path'], how='left')\n    \n    sub[['site_path_timestamp', 'floor', 'x', 'y']].to_csv('submission.csv', index=False)","metadata":{"papermill":{"duration":45.235595,"end_time":"2021-04-27T14:29:30.020482","exception":false,"start_time":"2021-04-27T14:28:44.784887","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}