{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Thanks for Kouki's sharing[LSTM by Keras with Unified Wi-Fi Feats](http://www.kaggle.com/kokitanisaka/lstm-by-keras-with-unified-wi-fi-feats), I just make a little change to it, I am used to writing codes with pytorch, So as you can see, I changed keras code to pytorch, socre 7.53**","metadata":{}},{"cell_type":"markdown","source":"## Overview\n\nIt demonstrats how to utilize [the unified Wi-Fi dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\nThe Neural Net model is not optimized, there's much space to improve the score. \n\nIn this notebook, I refer these two excellent notebooks.\n* [wifi features with lightgbm/KFold](https://www.kaggle.com/hiro5299834/wifi-features-with-lightgbm-kfold) by [@hiro5299834](https://www.kaggle.com/hiro5299834/)<br>\n I took some code fragments from his notebook.\n* [Simple 👌 99% Accurate Floor Model 💯](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model) by [@nigelhenry](https://www.kaggle.com/nigelhenry/)<br>\n I use his excellent work, the \"floor\" prediction.\n\nIt takes much much time to finish learning. <br>\nAnd even though I enable the GPU, it doesn't help. <br>\nIf anybody knows how to make it better, can you please make a comment? <br>\n\nThank you!","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\nfrom tqdm import tqdm\nimport random\nimport os\nimport copy\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nimport torch\nfrom torch import nn\nfrom torch.utils.data import Dataset, DataLoader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### options\nWe can change the way it learns with these options. <br>\nEspecialy **NUM_FEATS** is one of the most important options. <br>\nIt determines how many features are used in the training. <br>\nWe have 100 Wi-Fi features in the dataset, but 100th Wi-Fi signal sounds not important, right? <br>\nSo we can use top Wi-Fi signals if we think we need to. ","metadata":{}},{"cell_type":"code","source":"# options\n\nN_SPLITS = 10\n\nSEED = 2021\n\nNUM_FEATS = 49  # number of features that we use. there are 100 feats but we don't need to use all of them\n\nbase_path = '/kaggle'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\ndef get_timestamp():\n    import time\n    timestamp = ''\n    for i, d in enumerate(time.localtime()):\n        if i == 3:\n            d += 8\n        timestamp += str(d) + '-'\n        if i == 4:\n            break\n    return timestamp[:-1]\ndef comp_metric(xhat, yhat, fhat, x, y, f):\n    intermediate = np.sqrt((xhat-x)**2 + (yhat-y)**2) + 15 * np.abs(fhat-f)\n#     intermediate = np.sqrt((xhat-x)**2 + (yhat-y)**2)\n    return intermediate.sum()/xhat.shape[0]","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_dir = f\"{base_path}/input/indoorunifiedwifids\"\ntrain_files = sorted(glob.glob(os.path.join(feature_dir, '*_train.csv')))\ntest_files = sorted(glob.glob(os.path.join(feature_dir, '*_test.csv')))\nsubm = pd.read_csv(f'{base_path}/input/indoor-location-navigation/sample_submission.csv', index_col=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(f'{feature_dir}/train_all.pkl', 'rb') as f:\n  data = pickle.load( f)\n\nwith open(f'{feature_dir}/test_all.pkl', 'rb') as f:\n  test_data = pickle.load(f)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# training target features\nBSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS  = [f'rssi_{i}' for i in range(NUM_FEATS)]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get numbers of bssids to embed them in a layer\n\nwifi_bssids = []\nfor i in range(100):\n    wifi_bssids.extend(data.iloc[:,i].values.tolist())\nwifi_bssids = list(set(wifi_bssids))\n\nwifi_bssids_size = len(wifi_bssids)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids_test = []\nfor i in range(100):\n    wifi_bssids_test.extend(test_data.iloc[:,i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\n\nwifi_bssids_size = len(wifi_bssids_test)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids.extend(wifi_bssids_test)\nwifi_bssids_size = len(wifi_bssids)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# preprocess\n\nle = LabelEncoder()\nle.fit(wifi_bssids)\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\n\nss = StandardScaler()\nss.fit(data.loc[:,RSSI_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    data.loc[:,i] = le.transform(data.loc[:,i])\n    data.loc[:,i] = data.loc[:,i] + 1\n    \ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\n\ndata.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    test_data.loc[:,i] = le.transform(test_data.loc[:,i])\n    test_data.loc[:,i] = test_data.loc[:,i] + 1\n    \ntest_data.loc[:, 'site_id'] = le_site.transform(test_data.loc[:, 'site_id'])\n\ntest_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"site_count = len(data['site_id'].unique())\ndata.reset_index(drop=True, inplace=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set_seed(SEED)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The model\nThe first Embedding layer is very important. <br>\nThanks to the layer, we can make sense of these BSSID features. <br>\n<br>\nWe concatenate all the features and put them into LSTM. <br>\n<br>\nIf something is theoritically wrong, please correct me. Thank you in advance. ","metadata":{}},{"cell_type":"code","source":"class IndoorDataset(Dataset):\n    def __init__(self, data, flag='TRAIN'):\n        self.data = data\n        self.flag = flag\n    def __len__(self):\n        return self.data.shape[0]\n    def __getitem__(self, index):\n        tmp_data = self.data.iloc[index]\n        if self.flag == 'TRAIN':\n            ## 加载数据也许花费许久的时间\n            return {\n                'BSSID_FEATS':tmp_data[BSSID_FEATS].values.astype(float),\n                'RSSI_FEATS':tmp_data[RSSI_FEATS].values.astype(float),\n                'site_id':tmp_data['site_id'].astype(int),\n                'x':tmp_data['x'],\n                'y':tmp_data['y'],\n                'floor':tmp_data['floor'],\n            }\n        else:\n            return {\n                'BSSID_FEATS':tmp_data[BSSID_FEATS].values.astype(float),\n                'RSSI_FEATS':tmp_data[RSSI_FEATS].values.astype(float),\n                'site_id':tmp_data['site_id'].astype(int)\n            }\nclass simpleLSTM(nn.Module):\n    def __init__(self, embedding_dim = 32, seq_len=20):\n        super(simpleLSTM, self).__init__()\n        self.emb_BSSID_FEATS = nn.Embedding(wifi_bssids_size, embedding_dim)\n        self.emb_site_id = nn.Embedding(site_count, 16)\n        self.rssi_conv = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=1,\n                            out_channels=32,\n                            kernel_size=3,\n                            padding=1),\n                            nn.BatchNorm2d(32),\n                            torch.nn.ReLU())\n        self.conv1 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=80,\n                            out_channels=32,\n                            kernel_size=3,\n                            padding=1),\n                            nn.BatchNorm2d(32),\n                            torch.nn.ReLU())\n        self.conv2_3 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=32,\n                            out_channels=32,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(32))\n        self.conv4 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=64,\n                            out_channels=64,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(64),\n                            torch.nn.ReLU())\n        self.conv5_6 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=64,\n                            out_channels=32,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(32))\n        self.conv7 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=64,\n                            out_channels=64,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(64),\n                            torch.nn.ReLU())\n        self.conv8_9 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=64,\n                            out_channels=64,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(64))\n        self.conv10 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=128,\n                            out_channels=128,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(128),\n                            torch.nn.ReLU())\n        self.conv11_12 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=128,\n                            out_channels=64,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(64))\n        self.conv13 = torch.nn.Sequential(\n                        torch.nn.Conv2d(\n                            in_channels=128,\n                            out_channels=128,\n                            kernel_size=3,\n                            padding=2),\n                            nn.BatchNorm2d(128),\n                            torch.nn.ReLU())\n        self.fc1 = nn.Linear(67712, 256)\n        self.fc2 = nn.Linear(256, 2)\n        self.fc_floor1 = nn.Linear(67712, 128)\n        self.fc_floor2 = nn.Linear(128, 1)\n        self.batch_norm1 = nn.BatchNorm1d(NUM_FEATS)\n\n    def forward(self, x):\n        \n        x_site = x['site_id']\n        \n        x_site = torch.unsqueeze(x_site, -1)\n        x_site = torch.cat([x_site]*7, axis=1)\n        x_site = torch.unsqueeze(x_site, -1)\n        x_site = torch.cat([x_site]*7, axis=2)\n        x_site = self.emb_site_id(x_site)\n        x_site = x_site.permute(0,3,1,2)\n        \n        x_rssi = self.batch_norm1(x['RSSI_FEATS'])\n        x_rssi = torch.reshape(x_rssi, [-1,1,7,7])\n        x_rssi = self.rssi_conv(x_rssi)\n        x_bssid = x['BSSID_FEATS']\n        x_bssid = self.emb_BSSID_FEATS(x_bssid)\n        x_bssid = x_bssid.permute(0,2,1)\n        x_bssid = torch.reshape(x_bssid, [-1,32,7,7])\n        x = torch.cat([x_rssi,x_bssid,x_site],1)\n        # print(x.shape)\n        \n        x = self.conv1(x)\n        x_part1 = self.conv2_3(x)\n        x_part2 = self.conv2_3(x)\n        x = torch.cat([x_part1,x_part2],1)\n        x = self.conv4(x)\n        x_part1 = self.conv5_6(x)\n        x_part2 = self.conv5_6(x)\n        x = torch.cat([x_part1,x_part2],1)\n        x = self.conv7(x)\n        x_part1 = self.conv8_9(x)\n        x_part2 = self.conv8_9(x)\n        x = torch.cat([x_part1,x_part2],1)\n        x = self.conv10(x)\n        x_part1 = self.conv11_12(x)\n        x_part2 = self.conv11_12(x)\n        x = torch.cat([x_part1,x_part2],1)\n        x = self.conv13(x)\n        # print(x.shape)\n        x = torch.flatten(x, start_dim=1)\n        # print(x.shape)\n        \n        xy = self.fc1(x)\n        xy = self.fc2(xy)\n        floor = self.fc_floor1(x)\n        floor = self.fc_floor2(floor)\n        return xy, floor","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def evaluate(model, data_loader,  device='cuda'):\n    model.to(device)\n    model.eval()\n    x_list = []\n    y_list = []\n    floor_list = []\n    prexs_list = []\n    preys_list = []\n    prefloors_list = []\n    for d in tqdm(data_loader):\n        data_dict['BSSID_FEATS'] = d['BSSID_FEATS'].to(device).long()\n        data_dict['RSSI_FEATS'] = d['RSSI_FEATS'].to(device).float()\n        data_dict['site_id'] = d['site_id'].to(device).long()\n        x = d['x'].to(device).float()\n        y = d['y'].to(device).float()\n        floor = d['floor'].to(device).long()\n        x_list.append(x.cpu().detach().numpy())\n        y_list.append(y.cpu().detach().numpy())\n        floor_list.append(floor.cpu().detach().numpy())\n        xy, floor = model(data_dict)\n        prexs_list.append(xy[:, 0].cpu().detach().numpy())\n        preys_list.append(xy[:, 1].cpu().detach().numpy())\n        prefloors_list.append(floor.squeeze().cpu().detach().numpy())\n    x = np.concatenate(x_list)\n    y = np.concatenate(y_list)\n    floor = np.concatenate(floor_list)\n    prexs = np.concatenate(prexs_list)\n    preys =np.concatenate(preys_list)\n    prefloors = np.concatenate(prefloors_list)\n    eval_score = comp_metric(x, y, floor, prexs, preys, prefloors)\n    return eval_score\ndef get_result(model, data_loader, device='cuda'):\n    model.eval()\n    model.to(device)\n    prexs_list = []\n    preys_list = []\n    prefloors_list = []\n    data_dict = {}\n    for d in tqdm(data_loader):\n        data_dict['BSSID_FEATS'] = d['BSSID_FEATS'].to(device).long()\n        data_dict['RSSI_FEATS'] = d['RSSI_FEATS'].to(device).float()\n        data_dict['site_id'] = d['site_id'].to(device).long()\n        xy, floor = model(data_dict)\n        prexs_list.append(xy[:, 0].cpu().detach().numpy())\n        preys_list.append(xy[:, 1].cpu().detach().numpy())\n        prefloors_list.append(floor.squeeze(-1).cpu().detach().numpy())\n    prexs = np.concatenate(prexs_list)\n    preys =np.concatenate(preys_list)\n    prefloors = np.concatenate(prefloors_list)\n    return prexs, preys, prefloors","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_df = pd.DataFrame()\noof = list()\npredictions = list()\n\noof_x, oof_y, oof_f = np.zeros(data.shape[0]), np.zeros(data.shape[0]), np.zeros(data.shape[0])\npreds_x, preds_y = 0, 0\npreds_f_arr = np.zeros((test_data.shape[0], N_SPLITS))\n\nfor fold, (trn_idx, val_idx) in enumerate(StratifiedKFold(n_splits=N_SPLITS, shuffle=True, random_state=SEED).split(data.loc[:, 'path'], data.loc[:, 'path'])):\n    \n    train_data = data.loc[trn_idx]\n    valid_data = data.loc[val_idx]\n    train_dataset = IndoorDataset(train_data)\n    train_dataloader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=4)\n    valid_dataset = IndoorDataset(valid_data)\n    valid_dataloader = DataLoader(valid_dataset, batch_size=64, shuffle=True, num_workers=4)\n    test_dataset = IndoorDataset(test_data, 'TEST')\n    test_dataloader = DataLoader(test_dataset, batch_size=64, shuffle=False, num_workers=4)\n    device = 'cuda' if torch.cuda.is_available() else 'cpu'\n    model = simpleLSTM()\n    model = model.to(device)\n    \n    mse = nn.MSELoss()\n    mse = mse.to(device)\n    optim = torch.optim.Adam(model.parameters(), lr=1e-3)\n    \n    data_dict ={}\n    best_loss = 10000\n    num_epochs = 1\n    best_epoch = 0\n    for epoch in range(num_epochs):\n        model.train()\n        losses = []\n        pbar = tqdm(train_dataloader)\n        for d in pbar:\n            data_dict['BSSID_FEATS'] = d['BSSID_FEATS'].to(device).long()\n            data_dict['RSSI_FEATS'] = d['RSSI_FEATS'].to(device).float()\n            data_dict['site_id'] = d['site_id'].to(device).long()\n            x = d['x'].to(device).float().unsqueeze(-1)\n            y = d['y'].to(device).float().unsqueeze(-1)\n            floor = d['floor'].to(device).long()\n            xy, floor = model(data_dict)\n            label = torch.cat([x, y], dim=-1)\n            loss = mse(xy, label)\n            loss.backward()\n            optim.step()\n            optim.zero_grad()\n            losses.append(loss.cpu().detach().numpy())\n            pbar.set_description(f'loss:{np.mean(losses)}')\n        score = evaluate(model, valid_dataloader, device)\n        if score < best_loss:\n            best_loss = score\n            best_epoch = epoch\n            best_model = copy.deepcopy(model)\n        if best_epoch + 2<epoch:\n            break\n        print(\"*=\"*50)\n        print(f\"fold {fold} EPOCH {epoch}: mean position error {score}\")\n        print(\"*=\"*50)\n    test_x, test_y, test_floor = get_result(best_model, test_dataloader, device)\n    preds_f_arr[:,fold] = test_floor\n    preds_x += test_x\n    preds_y += test_y","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_x /= (fold + 1)\ntest_y /= (fold + 1)\n    \nprint(\"*+\"*40)\n# as it breaks in the middle of cross-validation, the score is not accurate at all.\nscore = comp_metric(oof_x, oof_y, oof_f, data.iloc[:, -5].to_numpy(), data.iloc[:, -4].to_numpy(), data.iloc[:, -3].to_numpy())\noof.append(score)\nprint(f\"mean position error {score}\")\nprint(\"*+\"*40)\n\npreds_f_mode = stats.mode(preds_f_arr, axis=1)\npreds_f = preds_f_mode[0].astype(int).reshape(-1)\ntest_preds = pd.DataFrame(np.stack((preds_f, test_x, test_y))).T\ntest_preds.columns = subm.columns\ntest_preds.index = test_data[\"site_path_timestamp\"]\ntest_preds[\"floor\"] = test_preds[\"floor\"].astype(int)\npredictions.append(test_preds)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_preds = pd.concat(predictions)\nall_preds = all_preds.reindex(subm.index)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fix the floor prediction\nSo far, it is not successfully make the \"floor\" prediction part with this dataset. <br>\nTo make it right, we can incorporate [@nigelhenry](https://www.kaggle.com/nigelhenry/)'s [excellent work](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model). <br>","metadata":{}},{"cell_type":"code","source":"simple_accurate_99 = pd.read_csv('../input/simple-99-accurate-floor-model/submission.csv')\nall_preds['floor'] = simple_accurate_99['floor'].values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_preds.to_csv('submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's it. \n\nThank you for reading all of it.\n\nI hope it helps!\n\nPlease make comments if you found something to point out, insights or suggestions. ","metadata":{}}]}