{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook, I share the method to get a more correct waypoint corresponding to wifi timestamp by linear interpolation.  \n\nIf there are some bug or my misunderstanding, please teach me!","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport json\nimport glob\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom scipy import interpolate","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Read sample data from published dataset","metadata":{}},{"cell_type":"markdown","source":"I use [a great dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids) by [@kouki](https://www.kaggle.com/kokitanisaka).\nThanks for sharing.","metadata":{}},{"cell_type":"code","source":"SAMPLE_SITE = '5a0546857ecc773753327266'\nSAMPLE_PATH = '5e158ecff4c3420006d52164'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get sample path dataframe from kouki's dataset.\nsample = pd.read_csv(f'../input/indoorunifiedwifids/{SAMPLE_SITE}_train.csv')\npath_df = sample[sample['path']==SAMPLE_PATH]\npath_df.loc[:, 'x':]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visalize waypoint\nplt.scatter(path_df['x'], path_df['y'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'number of wifi data: {len(path_df)}')\nprint(f'number of wifi unique data: {path_df[\"x\"].nunique()}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this sample path, there are 27 wifi data but unique wifi data are only 8, because this dataset are assigning the given waypoint to the closest wifi data.  \nThis approach is reasonable, but x,y corresponding to some wifi data are not correct values.  \nThen, I calculate a more correct waypoint corresponding to the wifi timestamp by linear interpolation.","metadata":{}},{"cell_type":"markdown","source":"## Read raw data","metadata":{}},{"cell_type":"code","source":"# https://github.com/location-competition/indoor-location-competition-20/blob/master/io_f.py\nfrom dataclasses import dataclass\n\n@dataclass\nclass ReadData:\n    acce: np.ndarray\n    acce_uncali: np.ndarray\n    gyro: np.ndarray\n    gyro_uncali: np.ndarray\n    magn: np.ndarray\n    magn_uncali: np.ndarray\n    ahrs: np.ndarray\n    wifi: np.ndarray\n    ibeacon: np.ndarray\n    waypoint: np.ndarray\n        \n\ndef read_data_file(data_filename):\n    acce = []\n    acce_uncali = []\n    gyro = []\n    gyro_uncali = []\n    magn = []\n    magn_uncali = []\n    ahrs = []\n    wifi = []\n    ibeacon = []\n    waypoint = []\n\n    with open(data_filename, 'r', encoding='utf-8') as file:\n        lines = file.readlines()\n\n    for line_data in lines:\n        line_data = line_data.strip()\n        if not line_data or line_data[0] == '#':\n            continue\n\n        line_data = line_data.split('\\t')\n\n        if line_data[1] == 'TYPE_ACCELEROMETER':\n            acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n            acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE':\n            gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n            gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n            magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n            magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ROTATION_VECTOR':\n            ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_WIFI':\n            sys_ts = line_data[0]\n            ssid = line_data[2]\n            bssid = line_data[3]\n            rssi = line_data[4]\n            lastseen_ts = line_data[6]\n            wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n            wifi.append(wifi_data)\n            continue\n\n        if line_data[1] == 'TYPE_BEACON':\n            ts = line_data[0]\n            uuid = line_data[2]\n            major = line_data[3]\n            minor = line_data[4]\n            rssi = line_data[6]\n            ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n            ibeacon.append(ibeacon_data)\n            continue\n\n        if line_data[1] == 'TYPE_WAYPOINT':\n            waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n\n    acce = np.array(acce)\n    acce_uncali = np.array(acce_uncali)\n    gyro = np.array(gyro)\n    gyro_uncali = np.array(gyro_uncali)\n    magn = np.array(magn)\n    magn_uncali = np.array(magn_uncali)\n    ahrs = np.array(ahrs)\n    wifi = np.array(wifi)\n    ibeacon = np.array(ibeacon)\n    waypoint = np.array(waypoint)\n\n    return ReadData(acce, acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs, wifi, ibeacon, waypoint)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read raw data \npath_file = glob.glob(f'../input/indoor-location-navigation/train/{SAMPLE_SITE}/*/{SAMPLE_PATH}.txt')[0]\nexample = read_data_file(path_file)\nfloor = path_file.split('/')[-2]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize waypoint(same as above)\ntrajectory = example.waypoint\ntrajectory_timestamp = trajectory[:,0]\ntrajectory_waypoint = trajectory[:, 1:]\nobserved_x = trajectory_waypoint[:, 0]\nobserved_y = trajectory_waypoint[:, 1]\n\nplt.scatter(observed_x, observed_y)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize with map\n# json\npath_json = f'../input/indoor-location-navigation/metadata/{SAMPLE_SITE}/{floor}/floor_info.json'\nwith open(path_json) as json_file:\n    json_data = json.load(json_file)\nwidth = json_data[\"map_info\"][\"width\"]\nheight = json_data[\"map_info\"][\"height\"]\n\n# image\npath_img = f'../input/indoor-location-navigation/metadata/{SAMPLE_SITE}/{floor}/floor_image.png'\nim = Image.open(path_img)\n\nfig, ax = plt.subplots(1, 1, figsize=(10, 10), dpi=100)\nax.imshow(np.asarray(im), extent=(0, width, 0, height))\nax.plot(observed_x, observed_y, linewidth=2, marker='o', markersize=7, color='red', label='observed')\nplt.legend()\nmargin=10\nax.set_xlim(observed_x.min()-margin, observed_x.max()+margin)\nax.set_ylim(observed_y.min()-margin, observed_y.max()+margin)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get wifi timestamp\nwifi = example.wifi\nwifi_timestamp = np.unique(wifi[:, 0]).astype(int)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Linear interpolation","metadata":{}},{"cell_type":"code","source":"def get_waypoint_by_linear_interpolation(\n    observed_timestamp: np.ndarray, \n    observed_x:np.ndarray, \n    observed_y:np.ndarray, \n    target_timestamp:np.ndarray, \n    delta_time=500\n    ):\n    \n    \"\"\"\n    observed: known Information\n    observed timesatmp, x and y are same shape\n    \"\"\"\n    target_waypoint_list = []\n    num_interpolation = len(observed_timestamp) - 1 \n    # \n    for i in range(num_interpolation):\n        # create latent timestamp\n        n_split = int((max(observed_timestamp[i:i+2]) - min(observed_timestamp[i:i+2])) / delta_time)\n        latent_timestamp = np.linspace(min(observed_timestamp[i:i+2]), max(observed_timestamp[i:i+2]), n_split).astype(int) \n        \n        # when x is ascending, latent is ascending too.\n        if observed_x[i] < observed_x[i+1]:\n            latent_x = np.linspace(min(observed_x[i:i+2]), max(observed_x[i:i+2]), n_split)\n        else:\n            latent_x = np.linspace(min(observed_x[i:i+2]), max(observed_x[i:i+2]), n_split)[::-1]\n        \n        # fitting\n        fitting_func = interpolate.interp1d(observed_x[i:i+2], observed_y[i:i+2])\n\n        target_x = []\n        target_y = []\n\n        # only \n        if i == num_interpolation-1:\n            target_idx = min(observed_timestamp[i:i+2]) <= target_timestamp\n        elif i == 0:\n            target_idx = target_timestamp < max(observed_timestamp[i:i+2])\n        else:\n            target_idx = (min(observed_timestamp[i:i+2]) <= target_timestamp) & (target_timestamp < max(observed_timestamp[i:i+2]))\n        target_use_timestamp = target_timestamp[target_idx]\n\n        # get the target waypoint with the closest timestamp\n        for t in target_use_timestamp:\n            idx = np.abs(latent_timestamp - t).argmin()\n            target_x.append(latent_x[idx])\n            \n            # although I don't know, there are nan sometimes.\n            if np.isnan(fitting_func(latent_x[idx])).sum() > 0:\n                # print('y has Nan')\n                idx = np.abs(observed_timestamp - t).argmin()\n                target_y.append(observed_y[idx])\n            else:\n                target_y.append(fitting_func(latent_x[idx]))\n\n        assert len(target_x) == len(target_y)\n        target_waypoint = np.stack([target_x, target_y], axis=1)\n        target_waypoint_list.append(target_waypoint)\n\n    target_waypoint = np.concatenate(target_waypoint_list)\n    return target_waypoint","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calc waypoint corresponding to wifi timestamp\nwifi_waypoint = get_waypoint_by_linear_interpolation(\n    observed_timestamp=trajectory_timestamp, \n    observed_x=observed_x, \n    observed_y=observed_y, \n    target_timestamp=wifi_timestamp, \n)\nwifi_waypoint","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Result","metadata":{}},{"cell_type":"code","source":"wifi_x = wifi_waypoint[:,0]\nwifi_y = wifi_waypoint[:,1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize observed waypoint(red) and interpolated wifi-waypoint(blue)\nfig, ax = plt.subplots(1, 1, figsize=(10, 10), dpi=100)\nax.imshow(np.asarray(im), extent=(0, width, 0, height))\nax.plot(observed_x, observed_y, linewidth=2, marker='o', alpha=1.0, markersize=7, color='red', label='observed')\nax.plot(wifi_x, wifi_y, linewidth=2, marker='o', alpha=0.5, markersize=7, color='blue', label='wifi')\nplt.legend()\nmargin=10\nax.set_xlim(wifi_x.min()-margin, wifi_x.max()+margin)\nax.set_ylim(wifi_y.min()-margin, wifi_y.max()+margin)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path_df['wifi_x'] = wifi_x\npath_df['wifi_y'] = wifi_y\npath_df.loc[:, 'x':]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"↑  \n  \nx,y: waypoint before interpolation  \nwifi_x,wifi_y: waypoint after interpolation  \n  \n**NOTE:** be careful to make sure the order of waypoint. I recommend to merge based on timestamp. ","metadata":{}},{"cell_type":"markdown","source":"The difference between before and after interpolation is as follows.","metadata":{}},{"cell_type":"code","source":"plt.hist(path_df['x'] - path_df['wifi_x'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.hist(path_df['y'] - path_df['wifi_y'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can interpolate the waypoint of all wifi data by applying the same process to other paths. Thanks!","metadata":{}}]}