{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Indoor Location - Exploratory Data Analysis\n\n\nQuick Exploratory Data Analysis for [Indoor Location & Navigation challenge](https://www.kaggle.com/c/indoor-location-navigation)\n\n\nIn this competition, your task is to predict the indoor position of smartphones based on real-time sensor data, provided by indoor positioning technology company XYZ10 in partnership with Microsoft Research. You'll locate devices using “active” localization data, which is made available with the cooperation of the user. Unlike passive localization methods (e.g. radar, camera), the data provided for this competition requires explicit user permission. You'll work with a dataset of nearly 30,000 traces from over 200 buildings."},{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/22559/logos/header.png)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"top\"></a>\n\n<div class=\"list-group\" id=\"list-tab\" role=\"tablist\">\n<h3 class=\"list-group-item list-group-item-action active\" data-toggle=\"list\" style='color:white; background:gray; border:0' role=\"tab\" aria-controls=\"home\"><center>Quick Navigation</center></h3>\n\n* [Metadata](#1)\n* [Train](#2)\n* [Test](#3)\n* [Read data](#4)\n* [Visualize data](#5)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a>\n<h2 style='background:gray; border:0; color:white'><center>Metadata<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"**metadata** - floor metadata folder, organized by site and floor, which includes the following for each floor:\n- floor_image.png\n- floor_info.json\n- geojson_map.json"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nimport glob\nimport math\nimport json\n\nfrom dataclasses import dataclass\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\nimport plotly.graph_objs as go\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_all_floors(path_to_building):\n    plt.figure(figsize=(16, 10))\n    floor_paths = glob.glob(os.path.join(path_to_building, \"*/floor_image.png\"))\n    for ind, floor_path in enumerate(floor_paths):\n        w = math.ceil(len(floor_paths) / 2)\n        h = math.ceil(len(floor_paths) / w)\n        if len(floor_paths) <= 4:\n            w = 4\n            h = 1\n        plt.subplot(h, w, ind + 1)\n        image = cv2.imread(floor_path)  \n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        plt.imshow(image)\n        plt.axis(\"off\")\n        plt.title(floor_path.split(\"/\")[-2], fontsize=16)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_all_floors(\"../input/indoor-location-navigation/metadata/5a0546857ecc773753327266/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_all_floors(\"../input/indoor-location-navigation/metadata/5c3c44b80379370013e0fd2b/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_all_floors(\"../input/indoor-location-navigation/metadata/5cd56b64e2acfd2d33b5932f/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_all_floors(\"../input/indoor-location-navigation/metadata/5cd56b6be2acfd2d33b59d1f/\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_all_floors(\"../input/indoor-location-navigation/metadata/5cd56baee2acfd2d33b61a93/\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n<h2 style='background:gray; border:0; color:white'><center>Train<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"**train** - training path files, organized by site and floor;    \neach path files contains the data of a single path on a single floor"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Number of train sites {len(os.listdir('../input/indoor-location-navigation/train/'))}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a>\n<h2 style='background:gray; border:0; color:white'><center>Test<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"**test** - test path files, organized by site and floor;    \neach path files contains the data of a single path on a single floor, but without the waypoint (x, y) data;    \nthe task of this competition is, for a given site-path file, predict the floor and waypoint locations at the timestamps given in the sample_submission.csv file"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Number of test sites {len(os.listdir('../input/indoor-location-navigation/test/'))}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**sample_submission.csv** - a sample submission file in the correct format;    \neach has a unique id which contains a site id, a path id, and the timestamp within the trace for which to make a prediction;     \nsee the [Evaluation page](https://www.kaggle.com/c/indoor-location-navigation/overview/evaluation) for the required integer mapping of floor names"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission = pd.read_csv(\"../input/indoor-location-navigation/sample_submission.csv\", index_col=0)\ndf_submission[\"x\"] = 100.\ndf_submission[\"y\"] = 96.\ndf_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_submission.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a>\n<h2 style='background:gray; border:0; color:white'><center>Read data<center><h2>"},{"metadata":{},"cell_type":"markdown","source":"Each train and test data file contains the next components:\n- acce (accelerometer)\n- acce_uncali\n- gyro (gyroscopes)\n- gyro_uncali\n- magn (magnetometer)\n- magn_uncali\n- ahrs (rotation vector)\n- wifi\n- ibeacon (bluetooth iBeacon)\n- waypoint (true location)\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# copy from https://github.com/location-competition/indoor-location-competition-20/blob/master/io_f.py\n\n\n@dataclass\nclass ReadData:\n    acce: np.ndarray\n    acce_uncali: np.ndarray\n    gyro: np.ndarray\n    gyro_uncali: np.ndarray\n    magn: np.ndarray\n    magn_uncali: np.ndarray\n    ahrs: np.ndarray\n    wifi: np.ndarray\n    ibeacon: np.ndarray\n    waypoint: np.ndarray\n\n\ndef read_data_file(data_filename):\n    acce = []\n    acce_uncali = []\n    gyro = []\n    gyro_uncali = []\n    magn = []\n    magn_uncali = []\n    ahrs = []\n    wifi = []\n    ibeacon = []\n    waypoint = []\n\n    with open(data_filename, 'r', encoding='utf-8') as file:\n        lines = file.readlines()\n\n    for line_data in lines:\n        line_data = line_data.strip()\n        if not line_data or line_data[0] == '#':\n            continue\n\n        line_data = line_data.split('\\t')\n\n        if line_data[1] == 'TYPE_ACCELEROMETER':\n            acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n            acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE':\n            gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n            gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n            magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n            magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ROTATION_VECTOR':\n            ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_WIFI':\n            sys_ts = line_data[0]\n            ssid = line_data[2]\n            bssid = line_data[3]\n            rssi = line_data[4]\n            lastseen_ts = line_data[6]\n            wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n            wifi.append(wifi_data)\n            continue\n\n        if line_data[1] == 'TYPE_BEACON':\n            ts = line_data[0]\n            uuid = line_data[2]\n            major = line_data[3]\n            minor = line_data[4]\n            rssi = line_data[6]\n            ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n            ibeacon.append(ibeacon_data)\n            continue\n\n        if line_data[1] == 'TYPE_WAYPOINT':\n            waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n\n    acce = np.array(acce)\n    acce_uncali = np.array(acce_uncali)\n    gyro = np.array(gyro)\n    gyro_uncali = np.array(gyro_uncali)\n    magn = np.array(magn)\n    magn_uncali = np.array(magn_uncali)\n    ahrs = np.array(ahrs)\n    wifi = np.array(wifi)\n    ibeacon = np.array(ibeacon)\n    waypoint = np.array(waypoint)\n    \n#     print(acce.shape)\n#     print(acce_uncali.shape)\n#     print(gyro.shape)\n#     print(gyro_uncali.shape)\n#     print(magn.shape)\n#     print(magn_uncali.shape)\n#     print(ahrs.shape)\n#     print(wifi.shape)\n#     print(ibeacon.shape)\n#     print(waypoint.shape)\n    \n    return ReadData(acce, acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs, wifi, ibeacon, waypoint)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_floor_data = read_data_file(\"../input/indoor-location-navigation/train/5a0546857ecc773753327266/B1/5e15730aa280850006f3d005.txt\")\nprint(\"acce\", train_floor_data.acce.shape)\nprint(\"acce_uncali\", train_floor_data.acce_uncali.shape)\nprint(\"gyro\", train_floor_data.gyro.shape)\nprint(\"gyro_uncali\", train_floor_data.gyro_uncali.shape)\nprint(\"magn\", train_floor_data.magn.shape)\nprint(\"magn_uncali\", train_floor_data.magn_uncali.shape)\nprint(\"ahrs\", train_floor_data.ahrs.shape)\nprint(\"wifi\", train_floor_data.wifi.shape)\nprint(\"ibeacon\", train_floor_data.ibeacon.shape)\nprint(\"waypoint\", train_floor_data.waypoint.shape)\nwith open(\"../input/indoor-location-navigation/metadata/5a0546857ecc773753327266/B1/floor_info.json\") as f:\n    train_floor_info = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_floor_data = read_data_file(\"../input/indoor-location-navigation/test/00ff0c9a71cc37a2ebdd0f05.txt\")\nprint(\"acce\", test_floor_data.acce.shape)\nprint(\"acce_uncali\", test_floor_data.acce_uncali.shape)\nprint(\"gyro\", test_floor_data.gyro.shape)\nprint(\"gyro_uncali\", test_floor_data.gyro_uncali.shape)\nprint(\"magn\", test_floor_data.magn.shape)\nprint(\"magn_uncali\", test_floor_data.magn_uncali.shape)\nprint(\"ahrs\", test_floor_data.ahrs.shape)\nprint(\"wifi\", test_floor_data.wifi.shape)\nprint(\"ibeacon\", test_floor_data.ibeacon.shape)\nprint(\"waypoint\", test_floor_data.waypoint.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a>\n<h2 style='background:gray; border:0; color:white'><center>Visualize data<center><h2>"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# copy from https://github.com/location-competition/indoor-location-competition-20/blob/master/visualize_f.py\n\n\ndef save_figure_to_html(fig, filename):\n    fig.write_html(filename)\n\n\ndef visualize_trajectory(trajectory, floor_plan_filename, width_meter, height_meter, title=None, mode='lines + markers + text', show=False):\n    fig = go.Figure()\n\n    # add trajectory\n    size_list = [6] * trajectory.shape[0]\n    size_list[0] = 10\n    size_list[-1] = 10\n\n    color_list = ['rgba(4, 174, 4, 0.5)'] * trajectory.shape[0]\n    color_list[0] = 'rgba(12, 5, 235, 1)'\n    color_list[-1] = 'rgba(235, 5, 5, 1)'\n\n    position_count = {}\n    text_list = []\n    for i in range(trajectory.shape[0]):\n        if str(trajectory[i]) in position_count:\n            position_count[str(trajectory[i])] += 1\n        else:\n            position_count[str(trajectory[i])] = 0\n        text_list.append('        ' * position_count[str(trajectory[i])] + f'{i}')\n    text_list[0] = 'Start Point: 0'\n    text_list[-1] = f'End Point: {trajectory.shape[0] - 1}'\n\n    fig.add_trace(\n        go.Scattergl(\n            x=trajectory[:, 0],\n            y=trajectory[:, 1],\n            mode=mode,\n            marker=dict(size=size_list, color=color_list),\n            line=dict(shape='linear', color='rgb(100, 10, 100)', width=2, dash='dot'),\n            text=text_list,\n            textposition=\"top center\",\n            name='trajectory',\n        ))\n\n    # add floor plan\n    floor_plan = Image.open(floor_plan_filename)\n    fig.update_layout(images=[\n        go.layout.Image(\n            source=floor_plan,\n            xref=\"x\",\n            yref=\"y\",\n            x=0,\n            y=height_meter,\n            sizex=width_meter,\n            sizey=height_meter,\n            sizing=\"contain\",\n            opacity=1,\n            layer=\"below\",\n        )\n    ])\n\n    # configure\n    fig.update_xaxes(autorange=False, range=[0, width_meter])\n    fig.update_yaxes(autorange=False, range=[0, height_meter], scaleanchor=\"x\", scaleratio=1)\n    fig.update_layout(\n        title=go.layout.Title(\n            text=title or \"No title.\",\n            xref=\"paper\",\n            x=0,\n        ),\n        autosize=True,\n        width=900,\n        height=200 + 900 * height_meter / width_meter,\n        template=\"plotly_white\",\n    )\n\n    if show:\n        fig.show()\n\n    return fig\n\n\ndef visualize_heatmap(position, value, floor_plan_filename, width_meter, height_meter, colorbar_title=\"colorbar\", title=None, show=False):\n    fig = go.Figure()\n\n    # add heat map\n    fig.add_trace(\n        go.Scatter(x=position[:, 0],\n                   y=position[:, 1],\n                   mode='markers',\n                   marker=dict(size=7,\n                               color=value,\n                               colorbar=dict(title=colorbar_title),\n                               colorscale=\"Rainbow\"),\n                   text=value,\n                   name=title))\n\n    # add floor plan\n    floor_plan = Image.open(floor_plan_filename)\n    fig.update_layout(images=[\n        go.layout.Image(\n            source=floor_plan,\n            xref=\"x\",\n            yref=\"y\",\n            x=0,\n            y=height_meter,\n            sizex=width_meter,\n            sizey=height_meter,\n            sizing=\"contain\",\n            opacity=1,\n            layer=\"below\",\n        )\n    ])\n\n    # configure\n    fig.update_xaxes(autorange=False, range=[0, width_meter])\n    fig.update_yaxes(autorange=False, range=[0, height_meter], scaleanchor=\"x\", scaleratio=1)\n    fig.update_layout(\n        title=go.layout.Title(\n            text=title or \"No title.\",\n            xref=\"paper\",\n            x=0,\n        ),\n        autosize=True,\n        width=900,\n        height=200 + 900 * height_meter / width_meter,\n        template=\"plotly_white\",\n    )\n\n    if show:\n        fig.show()\n\n    return fig","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def visualize_train_trajectory(path):\n    _id, floor = path.split(\"/\")[:2]\n    \n    train_floor_data = read_data_file(os.path.join(\"../input/indoor-location-navigation/train/\", path))\n    with open(f\"../input/indoor-location-navigation/metadata/{_id}/{floor}/floor_info.json\") as f:\n        train_floor_info = json.load(f)\n\n    return visualize_trajectory(\n        train_floor_data.waypoint[:, 1:3], \n        f\"../input/indoor-location-navigation/metadata/{_id}/{floor}/floor_image.png\",\n        train_floor_info[\"map_info\"][\"width\"], \n        train_floor_info[\"map_info\"][\"height\"],\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_train_trajectory(\"5a0546857ecc773753327266/B1/5e15730aa280850006f3d005.txt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_train_trajectory(\"5cd56b67e2acfd2d33b596bd/F2/5cf5dd5bd1378b0009268ad5.txt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_train_trajectory(\"5cd56b5ae2acfd2d33b5854a/B1/5d07424a1a69370008bc5de5.txt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_train_trajectory(\"5cd56bade2acfd2d33b61a61/F5/5cf77f57e23e6e000833f22a.txt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_train_trajectory(\"5dc8cea7659e181adb076a3f/B1/5dcf883e94e49000061256bc.txt\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_train_trajectory(\"5c3c44b80379370013e0fd2b/F5/5d0852a04a2bd40008d47332.txt\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}