{"cells":[{"metadata":{},"cell_type":"markdown","source":"<div>\n    <h1 align=\"center\"> Snap to Grid - Part(B)</h1></h1>\n    <h2 align=\"center\">Identify the position of a smartphone in a shopping mall</h2>\n    <h3 align=\"center\">By: Somayyeh Gholami & Mehran Kazeminia</h3>\n</div>"},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Description:"},{"metadata":{},"cell_type":"markdown","source":"### - In this notebook, we want to improve the score of our previous notebook \"Part A\". The address of our previous notebook is as follows.\n\nhttps://www.kaggle.com/mehrankazeminia/part-a-indoor-navigation-comparative-method\n\n### - We have used the following notebook codes in this notebook. Thanks again for sharing this great notebook.\n\nhttps://www.kaggle.com/robikscube/indoor-navigation-snap-to-grid-post-processing\n\n### - \"Data Visualization\" is of particular importance in this challenge. Because the location of the corridors is important :)\n\n### -  The reason we publish two notebooks (Part A, B) is that at the end of this challenge we want to look at the public scores and the private scores for each of the methods and examine the so-called \"Overfitting\".\n\n### =================================\n\n### - To get a better score, we changed the amount of \"threshold\" from five to eight in the second version.\n\n## >>> Good Luck <<<\n\n"},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# If you find this work useful, please don't forget upvoting :)"},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Import "},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nimport numpy as np\nimport pandas as pd \nimport matplotlib.pylab as plt\n\nfrom scipy.spatial.distance import cdist\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Helper Functions"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def split_col(df):\n    df = pd.concat([\n        df['site_path_timestamp'].str.split('_', expand=True) \\\n        .rename(columns={0:'site',\n                         1:'path',\n                         2:'timestamp'}),\n        df\n    ], axis=1).copy()\n    return df\n\nfloor_map = {\"B2\":-2, \"B1\":-1, \"F1\":0, \"F2\": 1, \"F3\":2,\n             \"F4\":3, \"F5\":4, \"F6\":5, \"F7\":6,\"F8\":7,\"F9\":8,\n             \"1F\":0, \"2F\":1, \"3F\":2, \"4F\":3, \"5F\":4, \"6F\":5,\n             \"7F\":6, \"8F\": 7, \"9F\":8}\n\n\ndef plot_preds(\n    site,\n    floorNo,\n    sub=None,\n    true_locs=None,\n    base=\"../input/indoor-location-navigation\",\n    show_train=True,\n    show_preds=True,\n    fix_labels=True,\n    map_floor=None\n):\n    \"\"\"\n    Plots predictions on floorplan map.\n    \n    map_floor : use a different floor's map\n    \"\"\"\n    if map_floor is None:\n        map_floor = floorNo\n    # Prepare width_meter & height_meter (taken from the .json file)\n    floor_plan_filename = f\"{base}/metadata/{site}/{map_floor}/floor_image.png\"\n    json_plan_filename = f\"{base}/metadata/{site}/{map_floor}/floor_info.json\"\n    with open(json_plan_filename) as json_file:\n        json_data = json.load(json_file)\n\n    width_meter = json_data[\"map_info\"][\"width\"]\n    height_meter = json_data[\"map_info\"][\"height\"]\n\n    floor_img = plt.imread(f\"{base}/metadata/{site}/{map_floor}/floor_image.png\")\n\n    fig, ax = plt.subplots(figsize=(12, 12))\n    plt.imshow(floor_img)\n\n    if show_train:\n        true_locs = true_locs.query('site == @site and floorNo == @map_floor').copy()\n        true_locs[\"x_\"] = true_locs[\"x\"] * floor_img.shape[0] / height_meter\n        true_locs[\"y_\"] = (\n            true_locs[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        true_locs.query(\"site == @site and floorNo == @map_floor\").groupby(\"path\").plot(\n            x=\"x_\",\n            y=\"y_\",\n            style=\"+\",\n            ax=ax,\n            label=\"train waypoint location\",\n            color=\"grey\",\n            alpha=0.5,\n        )\n\n    if show_preds:\n        sub = sub.query('site == @site and floorNo == @floorNo').copy()\n        sub[\"x_\"] = sub[\"x\"] * floor_img.shape[0] / height_meter\n        sub[\"y_\"] = (\n            sub[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        for path, path_data in sub.query(\n            \"site == @site and floorNo == @floorNo\"\n        ).groupby(\"path\"):\n            path_data.plot(\n                x=\"x_\",\n                y=\"y_\",\n                style=\".-\",\n                ax=ax,\n                title=f\"{site} - floor - {floorNo}\",\n                alpha=1,\n                label=path,\n            )\n    if fix_labels:\n        handles, labels = ax.get_legend_handles_labels()\n        by_label = dict(zip(labels, handles))\n        plt.legend(\n            by_label.values(), by_label.keys(), loc=\"center left\", bbox_to_anchor=(1, 0.5)\n        )\n    return fig, ax\n\ndef sub_process(sub, train_waypoints):\n    train_waypoints['isTrainWaypoint'] = True\n    sub = split_col(sub[['site_path_timestamp','floor','x','y']]).copy()\n    sub = sub.merge(train_waypoints[['site','floorNo','floor']].drop_duplicates(), how='left')\n    sub = sub.merge(\n        train_waypoints[['x','y','site','floor','isTrainWaypoint']].drop_duplicates(),\n        how='left',\n        on=['site','x','y','floor']\n             )\n    sub['isTrainWaypoint'] = sub['isTrainWaypoint'].fillna(False)\n    return sub.copy()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Data Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_waypoints = pd.read_csv('../input/indoor-location-train-waypoints/train_waypoints.csv')\nsub = sub_process(pd.read_csv('../input/part-a-indoor-navigation-comparative-method/submission.csv'),\n                 train_waypoints)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Find the closest \"grid\" point for each prediction."},{"metadata":{"trusted":true},"cell_type":"code","source":"def add_xy(df):\n    df['xy'] = [(x, y) for x,y in zip(df['x'], df['y'])]\n    return df\n\ndef closest_point(point, points):\n    \"\"\" Find closest point from a list of points. \"\"\"\n    return points[cdist([point], points).argmin()]\n\nsub = add_xy(sub)\ntrain_waypoints = add_xy(train_waypoints)\n\nds = []\nfor (site, myfloor), d in sub.groupby(['site','floor']):\n    true_floor_locs = train_waypoints.loc[(train_waypoints['floor'] == myfloor) &\n                                          (train_waypoints['site'] == site)] \\\n        .reset_index(drop=True)\n    if len(true_floor_locs) == 0:\n        print(f'Skipping {site} {myfloor}')\n        continue\n    d['matched_point'] = [closest_point(x, list(true_floor_locs['xy'])) for x in d['xy']]\n    d['x_'] = d['matched_point'].apply(lambda x: x[0])\n    d['y_'] = d['matched_point'].apply(lambda x: x[1])\n    ds.append(d)\n\nsub = pd.concat(ds)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Apply a Threshold and \"Snap to Grid\""},{"metadata":{"trusted":true},"cell_type":"code","source":"def snap_to_grid(sub, threshold):\n    \"\"\"\n    Snap to grid if within a threshold.\n    \n    x, y are the predicted points.\n    x_, y_ are the closest grid points.\n    _x_, _y_ are the new predictions after post processing.\n    \"\"\"\n    sub['_x_'] = sub['x']\n    sub['_y_'] = sub['y']\n    sub.loc[sub['dist'] < threshold, '_x_'] = sub.loc[sub['dist'] < threshold]['x_']\n    sub.loc[sub['dist'] < threshold, '_y_'] = sub.loc[sub['dist'] < threshold]['y_']\n    return sub.copy()\n\n# Calculate the distances\nsub['dist'] = np.sqrt( (sub.x-sub.x_)**2 + (sub.y-sub.y_)**2 )\n\nsub_pp = snap_to_grid(sub, threshold=8)\n\nsub_pp = sub_pp[['site_path_timestamp','floor','_x_','_y_','site','path','floorNo']] \\\n    .rename(columns={'_x_':'x', '_y_':'y'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Save Post Processed Submission."},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_pp[['site_path_timestamp','floor','x','y']] \\\n    .to_csv('submission_snap_to_grid.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"},{"metadata":{},"cell_type":"markdown","source":"# Data Visualization"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def split_col(df):\n    \"\"\"\n    Split submission site/path/timestamp into individual columns.\n    \"\"\"\n    df = pd.concat(\n        [\n            df[\"site_path_timestamp\"]\n            .str.split(\"_\", expand=True)\n            .rename(columns={0: \"site\", 1: \"path\", 2: \"timestamp\"}),\n            df,\n        ],\n        axis=1,\n    ).copy()\n    return df\n\n\ndef plot_preds(\n    site,\n    floorNo,\n    sub=None,\n    true_locs=None,\n    base=\"../input/indoor-location-navigation\",\n    show_train=True,\n    show_preds=True,\n):\n    \"\"\"\n    Plots predictions on floorplan map.\n    \"\"\"\n    # Prepare width_meter & height_meter (taken from the .json file)\n    floor_plan_filename = f\"{base}/metadata/{site}/{floorNo}/floor_image.png\"\n    json_plan_filename = f\"{base}/metadata/{site}/{floorNo}/floor_info.json\"\n    with open(json_plan_filename) as json_file:\n        json_data = json.load(json_file)\n\n    width_meter = json_data[\"map_info\"][\"width\"]\n    height_meter = json_data[\"map_info\"][\"height\"]\n\n    floor_img = plt.imread(f\"{base}/metadata/{site}/{floorNo}/floor_image.png\")\n\n    fig, ax = plt.subplots(figsize=(12, 12))\n    plt.imshow(floor_img)\n\n    if show_train:\n        true_locs[\"x_\"] = true_locs[\"x\"] * floor_img.shape[0] / height_meter\n        true_locs[\"y_\"] = (\n            true_locs[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        true_locs.query(\"site == @site and floorNo == @floorNo\").groupby(\"path\").plot(\n            x=\"x_\",\n            y=\"y_\",\n            style=\"+\",\n            ax=ax,\n            label=\"train waypoint location\",\n            color=\"grey\",\n            alpha=0.5,\n        )\n\n    if show_preds:\n        sub[\"x_\"] = sub[\"x\"] * floor_img.shape[0] / height_meter\n        sub[\"y_\"] = (\n            sub[\"y\"] * -1 * floor_img.shape[1] / width_meter\n        ) + floor_img.shape[0]\n        for path, path_data in sub.query(\n            \"site == @site and floorNo == @floorNo\"\n        ).groupby(\"path\"):\n            path_data.plot(\n                x=\"x_\",\n                y=\"y_\",\n                style=\".-\",\n                ax=ax,\n                title=f\"{site} - floor - {floorNo}\",\n                alpha=1,\n                label=path,\n            )\n    return fig, ax\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"sub = split_col(sub_pp[['site_path_timestamp','floor','x','y']])\n\ntrue_locs = pd.read_csv(\"../input/indoor-location-train-waypoints/train_waypoints.csv\")\n\n# Add floor No to sub file\nsub = sub.merge(true_locs[[\"site\", \"floor\", \"floorNo\"]].drop_duplicates())\n\n\nfor (site, floorNo), d in sub.groupby([\"site\", \"floorNo\"]):\n    fig, ax = plot_preds(site, floorNo, sub, true_locs)\n    # Remove duplicate labels\n    handles, labels = ax.get_legend_handles_labels()\n    by_label = dict(zip(labels, handles))\n    plt.legend(\n        by_label.values(), by_label.keys(), loc=\"center left\", bbox_to_anchor=(1, 0.5)\n    )\n    plt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-success\">  \n</div>"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}