{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **Thanks for Kouki's sharing[LSTM by Keras with Unified Wi-Fi Feats](http://www.kaggle.com/kokitanisaka/lstm-by-keras-with-unified-wi-fi-feats), I just make a little change to it, I am used to writing codes with pytorch, So as you can see, I changed keras code to pytorch, socre 7.53**"},{"metadata":{},"cell_type":"markdown","source":"## Overview\n\nIt demonstrats how to utilize [the unified Wi-Fi dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\nThe Neural Net model is not optimized, there's much space to improve the score. \n\nIn this notebook, I refer these two excellent notebooks.\n* [wifi features with lightgbm/KFold](https://www.kaggle.com/hiro5299834/wifi-features-with-lightgbm-kfold) by [@hiro5299834](https://www.kaggle.com/hiro5299834/)<br>\n I took some code fragments from his notebook.\n* [Simple 👌 99% Accurate Floor Model 💯](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model) by [@nigelhenry](https://www.kaggle.com/nigelhenry/)<br>\n I use his excellent work, the \"floor\" prediction.\n\nIt takes much much time to finish learning. <br>\nAnd even though I enable the GPU, it doesn't help. <br>\nIf anybody knows how to make it better, can you please make a comment? <br>\n\nThank you!"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\nfrom tqdm import tqdm\nimport random\nimport os\nimport copy\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nimport torch\nfrom torch import nn\nfrom torch.utils.data import Dataset, DataLoader","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### options\nWe can change the way it learns with these options. <br>\nEspecialy **NUM_FEATS** is one of the most important options. <br>\nIt determines how many features are used in the training. <br>\nWe have 100 Wi-Fi features in the dataset, but 100th Wi-Fi signal sounds not important, right? <br>\nSo we can use top Wi-Fi signals if we think we need to. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# options\n\nN_SPLITS = 10\n\nSEED = 2021\n\nNUM_FEATS = 20 # number of features that we use. there are 100 feats but we don't need to use all of them\n\nbase_path = '/kaggle'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\ndef get_timestamp():\n    import time\n    timestamp = ''\n    for i, d in enumerate(time.localtime()):\n        if i == 3:\n            d += 8\n        timestamp += str(d) + '-'\n        if i == 4:\n            break\n    return timestamp[:-1]\ndef comp_metric(xhat, yhat, fhat, x, y, f):\n    intermediate = np.sqrt((xhat-x)**2 + (yhat-y)**2) + 15 * np.abs(fhat-f)\n#     intermediate = np.sqrt((xhat-x)**2 + (yhat-y)**2)\n    return intermediate.sum()/xhat.shape[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_dir = f\"{base_path}/input/indoorunifiedwifids\"\ntrain_files = sorted(glob.glob(os.path.join(feature_dir, '*_train.csv')))\ntest_files = sorted(glob.glob(os.path.join(feature_dir, '*_test.csv')))\nsubm = pd.read_csv(f'{base_path}/input/indoor-location-navigation/sample_submission.csv', index_col=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(f'{feature_dir}/train_all.pkl', 'rb') as f:\n  data = pickle.load( f)\n\nwith open(f'{feature_dir}/test_all.pkl', 'rb') as f:\n  test_data = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# training target features\nBSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS  = [f'rssi_{i}' for i in range(NUM_FEATS)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get numbers of bssids to embed them in a layer\n\nwifi_bssids = []\nfor i in range(100):\n    wifi_bssids.extend(data.iloc[:,i].values.tolist())\nwifi_bssids = list(set(wifi_bssids))\n\nwifi_bssids_size = len(wifi_bssids)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids_test = []\nfor i in range(100):\n    wifi_bssids_test.extend(test_data.iloc[:,i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\n\nwifi_bssids_size = len(wifi_bssids_test)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids.extend(wifi_bssids_test)\nwifi_bssids_size = len(wifi_bssids)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# preprocess\n\nle = LabelEncoder()\nle.fit(wifi_bssids)\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\n\nss = StandardScaler()\nss.fit(data.loc[:,RSSI_FEATS])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    data.loc[:,i] = le.transform(data.loc[:,i])\n    data.loc[:,i] = data.loc[:,i] + 1\n    \ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\n\ndata.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    test_data.loc[:,i] = le.transform(test_data.loc[:,i])\n    test_data.loc[:,i] = test_data.loc[:,i] + 1\n    \ntest_data.loc[:, 'site_id'] = le_site.transform(test_data.loc[:, 'site_id'])\n\ntest_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"site_count = len(data['site_id'].unique())\ndata.reset_index(drop=True, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set_seed(SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The model\nThe first Embedding layer is very important. <br>\nThanks to the layer, we can make sense of these BSSID features. <br>\n<br>\nWe concatenate all the features and put them into LSTM. <br>\n<br>\nIf something is theoritically wrong, please correct me. Thank you in advance. "},{"metadata":{"trusted":true},"cell_type":"code","source":"class IndoorDataset(Dataset):\n    def __init__(self, data, flag='TRAIN'):\n        self.data = data\n        self.flag = flag\n    def __len__(self):\n        return self.data.shape[0]\n    def __getitem__(self, index):\n        tmp_data = self.data.iloc[index]\n        if self.flag == 'TRAIN':\n            ## 加载数据也许花费许久的时间\n            return {\n                'BSSID_FEATS':tmp_data[BSSID_FEATS].values.astype(float),\n                'RSSI_FEATS':tmp_data[RSSI_FEATS].values.astype(float),\n                'site_id':tmp_data['site_id'].astype(int),\n                'x':tmp_data['x'],\n                'y':tmp_data['y'],\n                'floor':tmp_data['floor'],\n            }\n        else:\n            return {\n                'BSSID_FEATS':tmp_data[BSSID_FEATS].values.astype(float),\n                'RSSI_FEATS':tmp_data[RSSI_FEATS].values.astype(float),\n                'site_id':tmp_data['site_id'].astype(int)\n            }\nclass simpleLSTM(nn.Module):\n    def __init__(self, embedding_dim = 64, seq_len=20):\n        super(simpleLSTM, self).__init__()\n        self.emb_BSSID_FEATS = nn.Embedding(wifi_bssids_size, embedding_dim)\n        self.emb_site_id = nn.Embedding(site_count, 2)\n        self.lstm1 = nn.LSTM(input_size=256,hidden_size=128, dropout=0.3, bidirectional=False)\n        self.lstm2 = nn.LSTM(input_size=128,hidden_size=16, dropout=0.1, bidirectional=False)\n        self.lr = nn.Linear(NUM_FEATS, NUM_FEATS * embedding_dim)\n        self.lr1 = nn.Linear(2562, 256)\n        self.lr_xy = nn.Linear(16, 2)\n        self.lr_floor = nn.Linear(16, 1)\n        self.batch_norm1 = nn.BatchNorm1d(NUM_FEATS)\n        self.batch_norm2 = nn.BatchNorm1d(2562)\n        self.batch_norm3 = nn.BatchNorm1d(1)\n        self.dropout = nn.Dropout(0.3)\n    def forward(self, x):\n        \n        x_bssid = self.emb_BSSID_FEATS(x['BSSID_FEATS'])\n        x_bssid = torch.flatten(x_bssid, start_dim=-2)\n        \n        x_site_id = self.emb_site_id(x['site_id'])\n        x_site_id = torch.flatten(x_site_id, start_dim=-1)\n        x_rssi = self.batch_norm1(x['RSSI_FEATS'])\n        x_rssi = self.lr(x_rssi)\n        x_rssi = torch.relu(x_rssi)\n        \n        x = torch.cat([x_bssid, x_site_id, x_rssi], dim=-1)\n        x = self.batch_norm2(x)\n        x = self.dropout(x)\n        x = torch.relu(self.lr1(x))\n\n        x = x.unsqueeze(-2)\n        x = self.batch_norm3(x)\n        x = x.transpose(0, 1)\n        x, _ = self.lstm1(x)\n        x = x.transpose(0, 1)\n        x = torch.relu(x)\n        x = x.transpose(0, 1)\n        x, _ = self.lstm2(x)\n        x = x.transpose(0, 1)\n        x = torch.relu(x)\n        xy = self.lr_xy(x)\n        floor = self.lr_floor(x)\n        floor = torch.relu(floor)\n        return xy.squeeze(-2), floor.squeeze(-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def evaluate(model, data_loader,  device='cuda'):\n    model.to(device)\n    model.eval()\n    x_list = []\n    y_list = []\n    floor_list = []\n    prexs_list = []\n    preys_list = []\n    prefloors_list = []\n    for d in tqdm(data_loader):\n        data_dict['BSSID_FEATS'] = d['BSSID_FEATS'].to(device).long()\n        data_dict['RSSI_FEATS'] = d['RSSI_FEATS'].to(device).float()\n        data_dict['site_id'] = d['site_id'].to(device).long()\n        x = d['x'].to(device).float()\n        y = d['y'].to(device).float()\n        floor = d['floor'].to(device).long()\n        x_list.append(x.cpu().detach().numpy())\n        y_list.append(y.cpu().detach().numpy())\n        floor_list.append(floor.cpu().detach().numpy())\n        xy, floor = model(data_dict)\n        prexs_list.append(xy[:, 0].cpu().detach().numpy())\n        preys_list.append(xy[:, 1].cpu().detach().numpy())\n        prefloors_list.append(floor.squeeze().cpu().detach().numpy())\n    x = np.concatenate(x_list)\n    y = np.concatenate(y_list)\n    floor = np.concatenate(floor_list)\n    prexs = np.concatenate(prexs_list)\n    preys =np.concatenate(preys_list)\n    prefloors = np.concatenate(prefloors_list)\n    eval_score = comp_metric(x, y, floor, prexs, preys, prefloors)\n    return eval_score\ndef get_result(model, data_loader, device='cuda'):\n    model.eval()\n    model.to(device)\n    prexs_list = []\n    preys_list = []\n    prefloors_list = []\n    data_dict = {}\n    for d in tqdm(data_loader):\n        data_dict['BSSID_FEATS'] = d['BSSID_FEATS'].to(device).long()\n        data_dict['RSSI_FEATS'] = d['RSSI_FEATS'].to(device).float()\n        data_dict['site_id'] = d['site_id'].to(device).long()\n        xy, floor = model(data_dict)\n        prexs_list.append(xy[:, 0].cpu().detach().numpy())\n        preys_list.append(xy[:, 1].cpu().detach().numpy())\n        prefloors_list.append(floor.squeeze(-1).cpu().detach().numpy())\n    prexs = np.concatenate(prexs_list)\n    preys =np.concatenate(preys_list)\n    prefloors = np.concatenate(prefloors_list)\n    return prexs, preys, prefloors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score_df = pd.DataFrame()\noof = list()\npredictions = list()\n\noof_x, oof_y, oof_f = np.zeros(data.shape[0]), np.zeros(data.shape[0]), np.zeros(data.shape[0])\npreds_x, preds_y = 0, 0\npreds_f_arr = np.zeros((test_data.shape[0], N_SPLITS))\n\n# for fold, (trn_idx, val_idx) in enumerate(StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED).split(data.loc[:, 'path'], data.loc[:, 'path'])):\n    \n#     train_data = data.loc[trn_idx]\n#     valid_data = data.loc[val_idx]\n#     train_dataset = IndoorDataset(train_data)\n#     train_dataloader = DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=6)\n#     valid_dataset = IndoorDataset(valid_data)\n#     valid_dataloader = DataLoader(valid_dataset, batch_size=128, shuffle=True, num_workers=6)\n#     test_dataset = IndoorDataset(test_data, 'TEST')\n#     test_dataloader = DataLoader(test_dataset, batch_size=128, shuffle=False, num_workers=6)\n#     device = 'cuda' if torch.cuda.is_available() else 'cpu'\n#     model = simpleLSTM()\n#     model = model.to(device)\n    \n#     mse = nn.MSELoss()\n#     mse = mse.to(device)\n#     optim = torch.optim.Adam(model.parameters(), lr=5e-3)\n    \n#     data_dict ={}\n#     best_loss = 1000\n#     num_epochs = 1\n#     best_epoch = 0\n#     for epoch in range(num_epochs):\n#         model.train()\n#         losses = []\n#         pbar = tqdm(train_dataloader)\n#         for d in pbar:\n#             data_dict['BSSID_FEATS'] = d['BSSID_FEATS'].to(device).long()\n#             data_dict['RSSI_FEATS'] = d['RSSI_FEATS'].to(device).float()\n#             data_dict['site_id'] = d['site_id'].to(device).long()\n#             x = d['x'].to(device).float().unsqueeze(-1)\n#             y = d['y'].to(device).float().unsqueeze(-1)\n#             floor = d['floor'].to(device).long()\n#             xy, floor = model(data_dict)\n#             label = torch.cat([x, y], dim=-1)\n#             loss = mse(xy, label)\n#             loss.backward()\n#             optim.step()\n#             optim.zero_grad()\n#             losses.append(loss.cpu().detach().numpy())\n#             pbar.set_description(f'loss:{np.mean(losses)}')\n#         score = evaluate(model, valid_dataloader, device)\n#         if score < best_loss:\n#             best_loss = score\n#             best_epoch = epoch\n#             best_model = copy.deepcopy(model)\n#         if best_epoch + 2<epoch:\n#             break\n#         print(\"*=\"*50)\n#         print(f\"fold {fold} EPOCH {epoch}: mean position error {score}\")\n#         print(\"*=\"*50)\n#     test_x, test_y, test_floor = get_result(best_model, test_dataloader, device)\n#     preds_f_arr[:,fold] = test_floor\n#     preds_x += test_x\n#     preds_y += test_y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_x /= (fold + 1)\n# test_y /= (fold + 1)\n    \n# print(\"*+\"*40)\n# # as it breaks in the middle of cross-validation, the score is not accurate at all.\n# score = comp_metric(oof_x, oof_y, oof_f, data.iloc[:, -5].to_numpy(), data.iloc[:, -4].to_numpy(), data.iloc[:, -3].to_numpy())\n# oof.append(score)\n# print(f\"mean position error {score}\")\n# print(\"*+\"*40)\n\n# preds_f_mode = stats.mode(preds_f_arr, axis=1)\n# preds_f = preds_f_mode[0].astype(int).reshape(-1)\n# test_preds = pd.DataFrame(np.stack((preds_f, test_x, test_y))).T\n# test_preds.columns = subm.columns\n# test_preds.index = test_data[\"site_path_timestamp\"]\n# test_preds[\"floor\"] = test_preds[\"floor\"].astype(int)\n# predictions.append(test_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# all_preds = pd.concat(predictions)\n# all_preds = all_preds.reindex(subm.index)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fix the floor prediction\nSo far, it is not successfully make the \"floor\" prediction part with this dataset. <br>\nTo make it right, we can incorporate [@nigelhenry](https://www.kaggle.com/nigelhenry/)'s [excellent work](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model). <br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# simple_accurate_99 = pd.read_csv('../input/simple-99-accurate-floor-model/submission.csv')\n\n# all_preds['floor'] = simple_accurate_99['floor'].values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# all_preds.to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's it. \n\nThank you for reading all of it.\n\nI hope it helps!\n\nPlease make comments if you found something to point out, insights or suggestions. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}