{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Overview\n\nThere have been two great notbooks about post processing.\nNamely :\n\n* ['Post-processing by Cost Minimization'](https://www.kaggle.com/saitodevel01/indoor-post-processing-by-cost-minimization)' by [Akio Skatio](https://www.kaggle.com/saitodevel01) that creates optimizes a cost function witch give the ideal prediction, by taking in the prediction from one source( yous) and using it in conjunction with elatborated Imu input data. \n\n* ['Snap to Grid\" Post Processing'](https://www.kaggle.com/robikscube/indoor-navigation-snap-to-grid-post-processing) by [Rob Mulla](https://www.kaggle.com/robikscube) witch well, snaps to the grid\n\nThese two great notebooks are really worth looking at individaully, and i am gratefull for them being publlished and shared.  \nThe first one contains libraries for the processing of IMu data that i was not aware of and has solved some problems for me, giving me better higher quality problems to solve. \n\n\nI wanted to give what little i could to contribute into improving peoples problems, and helping in my own way. \n\nI realised that there was no reason not to use them in **succession** and no doubt it's what many are doing, here is a ready to apply version of this so you won't have to struggle in stringing them toghther. \nOn this Ensembling method [notebook](https://www.kaggle.com/mehrankazeminia/part-a-indoor-navigation-comparative-method) this assembly of post processings improved the score by roughly 2.6 points\n\n\n**Two methods of easy use** are proposed:\n\n* **Fork** could just attache you file submission into the input data and then reference it in the appointed space below\n* **Copy** the two blocks of prepared code, importing the data needed as input (listed in the top right corner) \n\nTo submit you will then have to select the submission file from the table to the left of the output section of the notebook- this or yourse.\n\n\nThere was a fair amount of work and it wasn't all that straight forward, and there was a bit of debugging necessary, so \n> *if you decide to use it, upvote it !*\n\nthank you"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\n\nimport random\nimport os\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Insert You Sub File Here:** "},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('../input/part-a-indoor-navigation-comparative-method/submission.csv') \nsubi= sub.index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Or take the following blocks and import it's realtive data**"},{"metadata":{},"cell_type":"markdown","source":"Notebook N1 Cost Minimization"},{"metadata":{"trusted":true},"cell_type":"code","source":"!git clone --depth 1 https://github.com/location-competition/indoor-location-competition-20 indoor_location_competition_20\n!rm -rf indoor_location_competition_20/data\nimport multiprocessing\nimport numpy as np\nimport pandas as pd\nimport scipy.interpolate\nimport scipy.sparse\nfrom tqdm import tqdm\n\nfrom indoor_location_competition_20.io_f import read_data_file\nimport indoor_location_competition_20.compute_f as compute_f\nINPUT_PATH = '../input/indoor-location-navigation'\ndef compute_rel_positions(acce_datas, ahrs_datas):\n    step_timestamps, step_indexs, step_acce_max_mins = compute_f.compute_steps(acce_datas)\n    headings = compute_f.compute_headings(ahrs_datas)\n    stride_lengths = compute_f.compute_stride_length(step_acce_max_mins)\n    step_headings = compute_f.compute_step_heading(step_timestamps, headings)\n    rel_positions = compute_f.compute_rel_positions(stride_lengths, step_headings)\n    return rel_positions\ndef correct_path(args):\n    path, path_df = args\n    \n    T_ref  = path_df['timestamp'].values\n    xy_hat = path_df[['x', 'y']].values\n    \n    example = read_data_file(f'{INPUT_PATH}/test/{path}.txt')\n    rel_positions = compute_rel_positions(example.acce, example.ahrs)\n    if T_ref[-1] > rel_positions[-1, 0]:\n        rel_positions = [np.array([[0, 0, 0]]), rel_positions, np.array([[T_ref[-1], 0, 0]])]\n    else:\n        rel_positions = [np.array([[0, 0, 0]]), rel_positions]\n    rel_positions = np.concatenate(rel_positions)\n    \n    T_rel = rel_positions[:, 0]\n    delta_xy_hat = np.diff(scipy.interpolate.interp1d(T_rel, np.cumsum(rel_positions[:, 1:3], axis=0), axis=0)(T_ref), axis=0)\n\n    N = xy_hat.shape[0]\n    delta_t = np.diff(T_ref)\n    alpha = (8.1)**(-2) * np.ones(N)\n    beta  = (0.4 + 0.4 * 1e-3 * delta_t)**(-2)\n    A = scipy.sparse.spdiags(alpha, [0], N, N)\n    B = scipy.sparse.spdiags( beta, [0], N-1, N-1)\n    D = scipy.sparse.spdiags(np.stack([-np.ones(N), np.ones(N)]), [0, 1], N-1, N)\n\n    Q = A + (D.T @ B @ D)\n    c = (A @ xy_hat) + (D.T @ (B @ delta_xy_hat))\n    xy_star = scipy.sparse.linalg.spsolve(Q, c)\n\n    return pd.DataFrame({\n        'site_path_timestamp' : path_df['site_path_timestamp'],\n        'floor' : path_df['floor'],\n        'x' : xy_star[:, 0],\n        'y' : xy_star[:, 1],\n    })\n\n\n\ntmp = sub['site_path_timestamp'].apply(lambda s : pd.Series(s.split('_')))\nsub['site'] = tmp[0]\nsub['path'] = tmp[1]\nsub['timestamp'] = tmp[2].astype(float)\n\nprocesses = multiprocessing.cpu_count()\nwith multiprocessing.Pool(processes=processes) as pool:\n    dfs = pool.imap_unordered(correct_path, sub.groupby('path'))\n    dfs = tqdm(dfs)\n    dfs = list(dfs)\nsub = pd.concat(dfs).sort_values('site_path_timestamp')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Notbook N2 -Snap the gap"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport json\nimport matplotlib.pylab as plt\n\ndef split_col(df):\n    df = pd.concat([\n        df['site_path_timestamp'].str.split('_', expand=True) \\\n        .rename(columns={0:'site',\n                         1:'path',\n                         2:'timestamp'}),\n        df\n    ], axis=1).copy()\n    return df\n\nfloor_map = {\"B2\":-2, \"B1\":-1, \"F1\":0, \"F2\": 1, \"F3\":2,\n             \"F4\":3, \"F5\":4, \"F6\":5, \"F7\":6,\"F8\":7,\"F9\":8,\n             \"1F\":0, \"2F\":1, \"3F\":2, \"4F\":3, \"5F\":4, \"6F\":5,\n             \"7F\":6, \"8F\": 7, \"9F\":8}\n\n\ndef plot_preds(\n    site,\n    floorNo,\n    sub=None,\n    true_locs=None,\n    base=\"../input/indoor-location-navigation\",\n    show_train=True,\n    show_preds=True,\n    fix_labels=True,\n    map_floor=None\n):\n    \"\"\"\n    Plots predictions on floorplan map.\n    \n    map_floor : use a different floor's map\n    \"\"\"\n    if map_floor is None:\n        map_floor = floorNo\n    # Prepare width_meter & height_meter (taken from the .json file)\n    floor_plan_filename = f\"{base}/metadata/{site}/{map_floor}/floor_image.png\"\n    json_plan_filename = f\"{base}/metadata/{site}/{map_floor}/floor_info.json\"\n    with open(json_plan_filename) as json_file:\n        json_data = json.load(json_file)\n\n    width_meter = json_data[\"map_info\"][\"width\"]\n    height_meter = json_data[\"map_info\"][\"height\"]\n\n\n\n    if show_train:\n        true_locs = true_locs.query('site == @site and floorNo == @map_floor').copy()\n        true_locs[\"x_\"] = true_locs[\"x\"] * floor_img.shape[0] / height_meter\n        true_locs[\"y_\"] = (\n            true_locs[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        true_locs.query(\"site == @site and floorNo == @map_floor\").groupby(\"path\").plot(\n            x=\"x_\",\n            y=\"y_\",\n            style=\"+\",\n            ax=ax,\n            label=\"train waypoint location\",\n            color=\"grey\",\n            alpha=0.5,\n        )\n\n    if show_preds:\n        sub = sub.query('site == @site and floorNo == @floorNo').copy()\n        sub[\"x_\"] = sub[\"x\"] * floor_img.shape[0] / height_meter\n        sub[\"y_\"] = (\n            sub[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        for path, path_data in sub.query(\n            \"site == @site and floorNo == @floorNo\"\n        ).groupby(\"path\"):\n            path_data.plot(\n                x=\"x_\",\n                y=\"y_\",\n                style=\".-\",\n                ax=ax,\n                title=f\"{site} - floor - {floorNo}\",\n                alpha=1,\n                label=path,\n            )\n    if fix_labels:\n        handles, labels = ax.get_legend_handles_labels()\n        by_label = dict(zip(labels, handles))\n        plt.legend(\n            by_label.values(), by_label.keys(), loc=\"center left\", bbox_to_anchor=(1, 0.5)\n        )\n    return fig, ax\n\ndef sub_process(sub, train_waypoints):\n    train_waypoints['isTrainWaypoint'] = True\n    sub = split_col(sub[['site_path_timestamp','floor','x','y']]).copy()\n    sub = sub.merge(train_waypoints[['site','floorNo','floor']].drop_duplicates(), how='left')\n    sub = sub.merge(\n        train_waypoints[['x','y','site','floor','isTrainWaypoint']].drop_duplicates(),\n        how='left',\n        on=['site','x','y','floor']\n             )\n    sub['isTrainWaypoint'] = sub['isTrainWaypoint'].fillna(False)\n    return sub.copy()\n\ntrain_waypoints = pd.read_csv('../input/indoor-location-train-waypoints/train_waypoints.csv')\n# sub = sub_process(pd.read_csv('../input/indoor-location-train-waypoints/6.578LB_submission.csv'),\n#                  train_waypoints)\nsub = sub_process(sub, train_waypoints)\n\n\n\n\n\nfrom scipy.spatial.distance import cdist\n\ndef add_xy(df):\n    df['xy'] = [(x, y) for x,y in zip(df['x'], df['y'])]\n    return df\n\ndef closest_point(point, points):\n    \"\"\" Find closest point from a list of points. \"\"\"\n    return points[cdist([point], points).argmin()]\n\nsub = add_xy(sub)\ntrain_waypoints = add_xy(train_waypoints)\n\nds = []\nfor (site, myfloor), d in sub.groupby(['site','floor']):\n    true_floor_locs = train_waypoints.loc[(train_waypoints['floor'] == myfloor) &\n                                          (train_waypoints['site'] == site)] \\\n        .reset_index(drop=True)\n    if len(true_floor_locs) == 0:\n        print(f'Skipping {site} {myfloor}')\n        ds.append(d)\n        continue\n    d['matched_point'] = [closest_point(x, list(true_floor_locs['xy'])) for x in d['xy']]\n    d['x_'] = d['matched_point'].apply(lambda x: x[0])\n    d['y_'] = d['matched_point'].apply(lambda x: x[1])\n    ds.append(d)\n\nsub = pd.concat(ds)\n\n\ndef snap_to_grid(sub, threshold):\n    \"\"\"\n    Snap to grid if within a threshold.\n    \n    x, y are the predicted points.\n    x_, y_ are the closest grid points.\n    _x_, _y_ are the new predictions after post processing.\n    \"\"\"\n    sub['_x_'] = sub['x']\n    sub['_y_'] = sub['y']\n    sub.loc[sub['dist'] < threshold, '_x_'] = sub.loc[sub['dist'] < threshold]['x_']\n    sub.loc[sub['dist'] < threshold, '_y_'] = sub.loc[sub['dist'] < threshold]['y_']\n    return sub.copy()\n\n# Calculate the distances\nsub['dist'] = np.sqrt( (sub.x-sub.x_)**2 + (sub.y-sub.y_)**2 )\n\nsub_pp = snap_to_grid(sub, threshold=5)\n\nsub_pp = sub_pp[['site_path_timestamp','floor','_x_','_y_','site','path','floorNo']] \\\n    .rename(columns={'_x_':'x', '_y_':'y'})\n\nfinal_preds=sub_pp.drop(columns=['path','site','floorNo'])\n\nfinal_preds=final_preds.reindex(subi)\nfinal_preds.index=final_preds['site_path_timestamp']\nfinal_preds.drop(columns=['site_path_timestamp'],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_preds.to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's it. \n\n\nI hope it helps!\n\nPlease make comments if you found something to point out, insights or suggestions. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}