{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Overview\n\nIn this notebook, I'm going to try to make more 'WAY_POINTS' using host's github code.<br>\nThe motive is, to utilize [snap to grid](https://www.kaggle.com/robikscube/indoor-navigation-snap-to-grid-post-processing) method more effectively.<br>\nI think we can assume that there are more possible positions in the buildings other than waypoints from the training set. <br>\nIf we can find other positions, I thought we can have better result with [snap to grid](https://www.kaggle.com/robikscube/indoor-navigation-snap-to-grid-post-processing).<br>\n<br>\nThere is a function named 'calibrate_magnetic_wifi_ibeacon_to_position' in 'main.py'.<br>\nIt can make infered waypoints from the path files.<br>\n<br>\nActually for my models, it didn't contribute boosting my result. <br>\nBut as I felt it have possibility, I chose publishing this. <br>\n<br>\nIf a predicted position is not in a pathway, it can be modified with this. <br>\nAs you can see in the last of this notebook, it can make so many waypoints. <br>\nSo it is not feasible if the original prediction is not accurate enough, [snap to grid](https://www.kaggle.com/robikscube/indoor-navigation-snap-to-grid-post-processing) won't work well with this waypoints. <br>\n<br>\nThe result of this notebook, the calibrated waypoints is available [here](https://www.kaggle.com/kokitanisaka/calibrated-waypoints). <br>\n<br>\nIf you got any thoughts, kindly drop some comments. Thank you! "},{"metadata":{"trusted":true},"cell_type":"code","source":"import multiprocessing\nimport numpy as np\nimport pandas as pd\nimport scipy.interpolate\nimport scipy.sparse\nfrom tqdm import tqdm\n\nimport glob\nimport multiprocessing\nfrom multiprocessing import Pool","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Option\nAs it takes some time to make waypoints files, you can suppress it turning the flag to False. <br>\nIn that case, waypoints are read from the dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"MAKE_WAYPOINTS = False","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!git clone --depth 1 https://github.com/location-competition/indoor-location-competition-20 indoor_location_competition_20\n!rm -rf indoor_location_competition_20/data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from indoor_location_competition_20.io_f import read_data_file\nimport indoor_location_competition_20.compute_f as compute_f","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# this code is from host's github\n\ndef calibrate_magnetic_wifi_ibeacon_to_position(path_file_list, waypoints=None):\n    mwi_datas = {}\n    for path_filename in path_file_list:\n        print(f'Processing {path_filename}...')\n\n        path_datas = read_data_file(path_filename)\n        acce_datas = path_datas.acce\n        magn_datas = path_datas.magn\n        ahrs_datas = path_datas.ahrs\n        wifi_datas = path_datas.wifi\n        ibeacon_datas = path_datas.ibeacon\n        if waypoints is None:\n            posi_datas = path_datas.waypoint\n        else:\n            posi_datas = waypoints\n\n        step_positions = compute_f.compute_step_positions(acce_datas, ahrs_datas, posi_datas)\n\n        if wifi_datas.size != 0:\n            sep_tss = np.unique(wifi_datas[:, 0].astype(float))\n            wifi_datas_list = compute_f.split_ts_seq(wifi_datas, sep_tss)\n            for wifi_ds in wifi_datas_list:\n                diff = np.abs(step_positions[:, 0] - float(wifi_ds[0, 0]))\n                index = np.argmin(diff)\n                target_xy_key = tuple(step_positions[index, 1:3])\n                if target_xy_key in mwi_datas:\n                    mwi_datas[target_xy_key]['wifi'] = np.append(mwi_datas[target_xy_key]['wifi'], wifi_ds, axis=0)\n                else:\n                    mwi_datas[target_xy_key] = {\n                        'magnetic': np.zeros((0, 4)),\n                        'wifi': wifi_ds,\n                        'ibeacon': np.zeros((0, 3))\n                    }\n\n        if ibeacon_datas.size != 0:\n            sep_tss = np.unique(ibeacon_datas[:, 0].astype(float))\n            ibeacon_datas_list = compute_f.split_ts_seq(ibeacon_datas, sep_tss)\n            for ibeacon_ds in ibeacon_datas_list:\n                diff = np.abs(step_positions[:, 0] - float(ibeacon_ds[0, 0]))\n                index = np.argmin(diff)\n                target_xy_key = tuple(step_positions[index, 1:3])\n                if target_xy_key in mwi_datas:\n                    mwi_datas[target_xy_key]['ibeacon'] = np.append(mwi_datas[target_xy_key]['ibeacon'], ibeacon_ds, axis=0)\n                else:\n                    mwi_datas[target_xy_key] = {\n                        'magnetic': np.zeros((0, 4)),\n                        'wifi': np.zeros((0, 5)),\n                        'ibeacon': ibeacon_ds\n                    }\n\n        sep_tss = np.unique(magn_datas[:, 0].astype(float))\n        magn_datas_list = compute_f.split_ts_seq(magn_datas, sep_tss)\n        for magn_ds in magn_datas_list:\n            diff = np.abs(step_positions[:, 0] - float(magn_ds[0, 0]))\n            index = np.argmin(diff)\n            target_xy_key = tuple(step_positions[index, 1:3])\n            if target_xy_key in mwi_datas:\n                mwi_datas[target_xy_key]['magnetic'] = np.append(mwi_datas[target_xy_key]['magnetic'], magn_ds, axis=0)\n            else:\n                mwi_datas[target_xy_key] = {\n                    'magnetic': magn_ds,\n                    'wifi': np.zeros((0, 5)),\n                    'ibeacon': np.zeros((0, 3))\n                }\n\n    return mwi_datas","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# this code is from host's github\n\ndef extract_magnetic_strength(mwi_datas):\n    magnetic_strength = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        magnetic_data = mwi_datas[position_key]['magnetic']\n        magnetic_s = np.mean(np.sqrt(np.sum(magnetic_data[:, 1:4] ** 2, axis=1)))\n        magnetic_strength[position_key] = magnetic_s\n\n    return magnetic_strength\n\n\ndef extract_wifi_rssi(mwi_datas):\n    wifi_rssi = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        wifi_data = mwi_datas[position_key]['wifi']\n        for wifi_d in wifi_data:\n            bssid = wifi_d[2]\n            rssi = int(wifi_d[3])\n\n            if bssid in wifi_rssi:\n                position_rssi = wifi_rssi[bssid]\n                if position_key in position_rssi:\n                    old_rssi = position_rssi[position_key][0]\n                    old_count = position_rssi[position_key][1]\n                    position_rssi[position_key][0] = (old_rssi * old_count + rssi) / (old_count + 1)\n                    position_rssi[position_key][1] = old_count + 1\n                else:\n                    position_rssi[position_key] = np.array([rssi, 1])\n            else:\n                position_rssi = {}\n                position_rssi[position_key] = np.array([rssi, 1])\n\n            wifi_rssi[bssid] = position_rssi\n\n    return wifi_rssi\n\n\ndef extract_ibeacon_rssi(mwi_datas):\n    ibeacon_rssi = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        ibeacon_data = mwi_datas[position_key]['ibeacon']\n        for ibeacon_d in ibeacon_data:\n            ummid = ibeacon_d[1]\n            rssi = int(ibeacon_d[2])\n\n            if ummid in ibeacon_rssi:\n                position_rssi = ibeacon_rssi[ummid]\n                if position_key in position_rssi:\n                    old_rssi = position_rssi[position_key][0]\n                    old_count = position_rssi[position_key][1]\n                    position_rssi[position_key][0] = (old_rssi * old_count + rssi) / (old_count + 1)\n                    position_rssi[position_key][1] = old_count + 1\n                else:\n                    position_rssi[position_key] = np.array([rssi, 1])\n            else:\n                position_rssi = {}\n                position_rssi[position_key] = np.array([rssi, 1])\n\n            ibeacon_rssi[ummid] = position_rssi\n\n    return ibeacon_rssi\n\n\ndef extract_wifi_count(mwi_datas):\n    wifi_counts = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        wifi_data = mwi_datas[position_key]['wifi']\n        count = np.unique(wifi_data[:, 2]).shape[0]\n        wifi_counts[position_key] = count\n\n    return wifi_counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"floor_map = {\"B2\":-2, \"B1\":-1, \"F1\":0, \"F2\": 1, \"F3\":2,\n             \"F4\":3, \"F5\":4, \"F6\":5, \"F7\":6,\"F8\":7,\"F9\":8,\n             \"1F\":0, \"2F\":1, \"3F\":2, \"4F\":3, \"5F\":4, \"6F\":5,\n             \"7F\":6, \"8F\": 7, \"9F\":8}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# target only buildings in test set\ndef generate_target_buildings():\n    ssubm = pd.read_csv(\n        '/kaggle/input/indoor-location-navigation/sample_submission.csv')\n    ssubm_df = ssubm[\"site_path_timestamp\"].apply(\n        lambda x: pd.Series(x.split(\"_\")))\n    return sorted(ssubm_df[0].value_counts().index.tolist()) # type: ignore\n\ntarget_buildings = generate_target_buildings()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function yield the waypoints files. \n\ndef process_waypoints(target_building):\n    floors = glob.glob(f'../input/indoor-location-navigation/train/{target_building}/*')\n    \n    waypoints = None\n    \n    for floor in floors:\n        floor_name = floor.split('/')[-1]\n        \n        floor_val = floor_map[floor_name]\n\n        paths = glob.glob(f'{floor}/*.txt')\n\n\n        for path in paths:\n            calibration = calibrate_magnetic_wifi_ibeacon_to_position([path])\n\n            path_name = path.split('/')[-1]\n\n            try:\n                my_dict = extract_wifi_count(calibration)\n\n                wifi_count = pd.DataFrame(list(my_dict.items()),columns = ['xy','wifi_count']) \n\n                wifi_count['type_name'] = 'TYPE_WAYPOINT'\n                wifi_count['x'] = wifi_count['xy'].apply(lambda x: x[0])\n                wifi_count['y'] = wifi_count['xy'].apply(lambda x: x[1])\n                wifi_count['timestamp'] = 0\n                wifi_count['site'] = target_building\n                wifi_count['floorNo'] = floor_name\n                wifi_count['floor'] = floor_val\n                wifi_count['path'] = path_name.replace('.txt', '')\n\n                if waypoints is None:\n                    waypoints = wifi_count\n                else:\n                    waypoints = pd.concat([waypoints, wifi_count])\n            except:\n                print(f' --- ERROR --- Building:{target_building} Floor:{floor_name} Path:{path_name}')\n                import traceback\n                traceback.print_exc()        \n    \n    waypoints[['type_name','x','y','timestamp','site','floorNo','floor','path']].to_csv(f'{target_building}_waypoints.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"num_cores = multiprocessing.cpu_count()\n\nif MAKE_WAYPOINTS:\n    with Pool(num_cores) as pool:\n        pool.map(process_waypoints, [t for t in target_buildings])  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport json\nimport matplotlib.pylab as plt\n\ndef split_col(df):\n    df = pd.concat([\n        df['site_path_timestamp'].str.split('_', expand=True) \\\n        .rename(columns={0:'site',\n                         1:'path',\n                         2:'timestamp'}),\n        df\n    ], axis=1).copy()\n    return df\n\nfloor_map = {\"B2\":-2, \"B1\":-1, \"F1\":0, \"F2\": 1, \"F3\":2,\n             \"F4\":3, \"F5\":4, \"F6\":5, \"F7\":6,\"F8\":7,\"F9\":8,\n             \"1F\":0, \"2F\":1, \"3F\":2, \"4F\":3, \"5F\":4, \"6F\":5,\n             \"7F\":6, \"8F\": 7, \"9F\":8}\n\n\ndef plot_preds(\n    site,\n    floorNo,\n    sub=None,\n    true_locs=None,\n    base=\"../input/indoor-location-navigation\",\n    show_train=True,\n    show_preds=True,\n    fix_labels=True,\n    map_floor=None\n):\n    \"\"\"\n    Plots predictions on floorplan map.\n    \n    map_floor : use a different floor's map\n    \"\"\"\n    if map_floor is None:\n        map_floor = floorNo\n    # Prepare width_meter & height_meter (taken from the .json file)\n    floor_plan_filename = f\"{base}/metadata/{site}/{map_floor}/floor_image.png\"\n    json_plan_filename = f\"{base}/metadata/{site}/{map_floor}/floor_info.json\"\n    with open(json_plan_filename) as json_file:\n        json_data = json.load(json_file)\n\n    width_meter = json_data[\"map_info\"][\"width\"]\n    height_meter = json_data[\"map_info\"][\"height\"]\n\n    floor_img = plt.imread(f\"{base}/metadata/{site}/{map_floor}/floor_image.png\")\n\n    fig, ax = plt.subplots(figsize=(12, 12))\n    plt.imshow(floor_img)\n\n    if show_train:\n        true_locs = true_locs.query('site == @site and floorNo == @map_floor').copy()\n        true_locs[\"x_\"] = true_locs[\"x\"] * floor_img.shape[0] / height_meter\n        true_locs[\"y_\"] = (\n            true_locs[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        true_locs.query(\"site == @site and floorNo == @map_floor\").groupby(\"path\").plot(\n            x=\"x_\",\n            y=\"y_\",\n            style=\"+\",\n            ax=ax,\n            label=\"train waypoint location\",\n            color=\"grey\",\n            alpha=0.5,\n        )\n\n    if show_preds:\n        sub = sub.query('site == @site and floorNo == @floorNo').copy()\n        sub[\"x_\"] = sub[\"x\"] * floor_img.shape[0] / height_meter\n        sub[\"y_\"] = (\n            sub[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        for path, path_data in sub.query(\n            \"site == @site and floorNo == @floorNo\"\n        ).groupby(\"path\"):\n            path_data.plot(\n                x=\"x_\",\n                y=\"y_\",\n                style=\".-\",\n                ax=ax,\n                title=f\"{site} - floor - {floorNo}\",\n                alpha=1,\n                label=path,\n            )\n    if fix_labels:\n        handles, labels = ax.get_legend_handles_labels()\n        by_label = dict(zip(labels, handles))\n        plt.legend(\n            by_label.values(), by_label.keys(), loc=\"center left\", bbox_to_anchor=(1, 0.5)\n        )\n    return fig, ax\n\ndef sub_process(sub, train_waypoints):\n    train_waypoints['isTrainWaypoint'] = True\n    sub = split_col(sub[['site_path_timestamp','floor','x','y']]).copy()\n    sub = sub.merge(train_waypoints[['site','floorNo','floor']].drop_duplicates(), how='left')\n    sub = sub.merge(\n        train_waypoints[['x','y','site','floor','isTrainWaypoint']].drop_duplicates(),\n        how='left',\n        on=['site','x','y','floor']\n             )\n    sub['isTrainWaypoint'] = sub['isTrainWaypoint'].fillna(False)\n    return sub.copy()\n\nfrom scipy.spatial.distance import cdist\n\ndef add_xy(df):\n    df['xy'] = [(x, y) for x,y in zip(df['x'], df['y'])]\n    return df\n\ndef closest_point(point, points):\n    \"\"\" Find closest point from a list of points. \"\"\"\n    return points[cdist([point], points).argmin()]\n\ndef snap_to_grid(sub, threshold):\n    \"\"\"\n    Snap to grid if within a threshold.\n    \n    x, y are the predicted points.\n    x_, y_ are the closest grid points.\n    _x_, _y_ are the new predictions after post processing.\n    \"\"\"\n    sub['_x_'] = sub['x']\n    sub['_y_'] = sub['y']\n    sub.loc[sub['dist'] < threshold, '_x_'] = sub.loc[sub['dist'] < threshold]['x_']\n    sub.loc[sub['dist'] < threshold, '_y_'] = sub.loc[sub['dist'] < threshold]['y_']\n    return sub.copy()\n\ndef snap(sub, train_waypoints, threshold=0):\n    #train_waypoints = pd.read_csv('../input/indoor-location-train-waypoints/train_waypoints.csv')\n\n    sub = sub_process(sub, train_waypoints)\n    \n    sub = add_xy(sub)\n    train_waypoints = add_xy(train_waypoints)\n\n    ds = []\n    for (site, myfloor), d in sub.groupby(['site','floor']):\n        true_floor_locs = train_waypoints.loc[(train_waypoints['floor'] == myfloor) &\n                                              (train_waypoints['site'] == site)] \\\n            .reset_index(drop=True)\n        if len(true_floor_locs) == 0:\n            print(f'Skipping {site} {myfloor}')\n            continue\n        d['matched_point'] = [closest_point(x, list(true_floor_locs['xy'])) for x in d['xy']]\n        d['x_'] = d['matched_point'].apply(lambda x: x[0])\n        d['y_'] = d['matched_point'].apply(lambda x: x[1])\n        ds.append(d)\n\n    sub = pd.concat(ds)\n    \n    # Calculate the distances\n    sub['dist'] = np.sqrt( (sub.x-sub.x_)**2 + (sub.y-sub.y_)**2 )\n\n    sub_pp = snap_to_grid(sub, threshold=threshold)\n\n    sub_pp = sub_pp[['site_path_timestamp','floor','_x_','_y_','site','path','floorNo']] \\\n        .rename(columns={'_x_':'x', '_y_':'y'})\n    \n    plot_preds(example_site, example_floorNo, sub_pp,\n               train_waypoints, show_preds=True)\n    plt.show()\n    \n    return sub_pp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if MAKE_WAYPOINTS:\n    waypoint_files = glob.glob('*_waypoints.csv')\nelse:\n    waypoint_files = glob.glob('../input/calibrated-waypoints/*_waypoints.csv')\n\ntrain_waypoints = None\nfor waypoint_file in waypoint_files:\n    wp = pd.read_csv(waypoint_file)\n    if train_waypoints is None:\n        train_waypoints = wp\n    else:\n        train_waypoints = pd.concat([train_waypoints, wp])\n        \noriginal_wp = pd.read_csv('../input/indoor-location-train-waypoints/train_waypoints.csv')\n        \ntrain_waypoints = pd.concat([train_waypoints, original_wp])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv('../input/indoor-location-navigation/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_site = '5dbc1d84c1eb61796cf7c010'\nexample_floorNo = 'F3'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The result \nLet's take a look the generated waypoints. <br>\nWe clearly see that we got many other waypoints. <br>\nCan be used, or not? <br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"snap(sample_submission, original_wp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"snap(sample_submission, train_waypoints)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}