{"cells":[{"metadata":{},"cell_type":"markdown","source":"## <div class=\"alert alert-block alert-warning\">There are some HTML visualizations. It takes some seconds, so please wait!⌛</div> "},{"metadata":{},"cell_type":"markdown","source":"# Get understand dataset by run processing scripts\n\n## Contents\n\n1. [Introduction](#1)\n1. [Preparete libraries](#2)\n1. [Setting](#3)\n1. [Run the tool](#4)\n1. [Check output data](#5)\n1. [How script extracts data from trace path file?](#6)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n# <div class=\"alert alert-block alert-info\">Introduction</div>"},{"metadata":{},"cell_type":"markdown","source":"## About this notebook\n\nFor processing and visualization of datas from android smartphone, [indoor-location-competition-20](https://github.com/location-competition/indoor-location-competition-20) repository is introduced in [Data page](https://www.kaggle.com/c/indoor-location-navigation/data). \n\nThis repository contains sample data and code for Indoor Location Competition 2.0.\n\nI have made this sample code work on kaggle notebook, so I'll share the result. And I also aim to get better understanding of the given data through setting up and running the tool."},{"metadata":{},"cell_type":"markdown","source":"### <u>Note</u>\n\nindoor-location-competition-20 is under MIT License. Copyright (c) 2017-2020 XYZ10, Inc. https://dangwu.com/\n\nTo check detail, please check [here](https://github.com/location-competition/indoor-location-competition-20/blob/master/LICENSE)."},{"metadata":{},"cell_type":"markdown","source":"## About Indoor Location Competition 2.0\n\n[Indoor Location Competition 2.0](https://location20.xyz10.com/) is a continuation of Microsoft Indoor Location Competition.\n\nWebinar Video of Indoor Location Competition 2.0 is here."},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('xt3OzMC-XMU')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## About sample processing scripts\n\nIndoor traces data processing scripts witch have been published in [indoor-location-competition-20](https://github.com/location-competition/indoor-location-competition-20) repository. These scripts has following five function.\n\n- Ground truth location visualization\n\n- Sample step detection and visualization\n\n- Geo-magnetic field intensity visualization\n\n- WiFi RSSI ([Received signal strength indication](https://en.wikipedia.org/wiki/Received_signal_strength_indication)) heatmap generation\n\n- iBeacon RSSI heatmap generation\n\n- WiFi SSID counts visualization\n\nIf you want to see sample visualizations, please jump [here](#5) !"},{"metadata":{},"cell_type":"markdown","source":"-------------------------------"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n# <div class=\"alert alert-block alert-info\">Preparete libraries</div>\n\nLoad libraries and clone tool."},{"metadata":{"trusted":true},"cell_type":"code","source":"import glob\nimport json\nimport os\nfrom pathlib import Path\nimport sys\n\n\nfrom IPython.display import IFrame\nimport numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!git clone -b develop https://github.com/tasotasoso/indoor-location-competition-20.git","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is what the repository looks like."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls indoor-location-competition-20","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By running main.py, we can process the data. However, since the data path is hardcoded and main.py asks for standard input, I will redefine only main.py in this jupyter notebook.\n\nAdd path for indoor-location-competition-20 and import modules Other than the main.py."},{"metadata":{"trusted":true},"cell_type":"code","source":"sys.path.append('indoor-location-competition-20')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from compute_f import split_ts_seq, compute_step_positions\nfrom io_f import read_data_file\nfrom visualize_f import visualize_trajectory, visualize_heatmap, save_figure_to_html","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I copied the functions in main.py. If you are interested, please open the cell or check the [repository](https://github.com/location-competition/indoor-location-competition-20)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def calibrate_magnetic_wifi_ibeacon_to_position(path_file_list):\n    mwi_datas = {}\n    for path_filename in path_file_list:\n        print(f'Processing {path_filename}...')\n\n        path_datas = read_data_file(path_filename)\n        acce_datas = path_datas.acce\n        magn_datas = path_datas.magn\n        ahrs_datas = path_datas.ahrs\n        wifi_datas = path_datas.wifi\n        ibeacon_datas = path_datas.ibeacon\n        posi_datas = path_datas.waypoint\n\n        step_positions = compute_step_positions(acce_datas, ahrs_datas, posi_datas)\n        # visualize_trajectory(posi_datas[:, 1:3], floor_plan_filename, width_meter, height_meter, title='Ground Truth', show=True)\n        # visualize_trajectory(step_positions[:, 1:3], floor_plan_filename, width_meter, height_meter, title='Step Position', show=True)\n\n        if wifi_datas.size != 0:\n            sep_tss = np.unique(wifi_datas[:, 0].astype(float))\n            wifi_datas_list = split_ts_seq(wifi_datas, sep_tss)\n            for wifi_ds in wifi_datas_list:\n                diff = np.abs(step_positions[:, 0] - float(wifi_ds[0, 0]))\n                index = np.argmin(diff)\n                target_xy_key = tuple(step_positions[index, 1:3])\n                if target_xy_key in mwi_datas:\n                    mwi_datas[target_xy_key]['wifi'] = np.append(mwi_datas[target_xy_key]['wifi'], wifi_ds, axis=0)\n                else:\n                    mwi_datas[target_xy_key] = {\n                        'magnetic': np.zeros((0, 4)),\n                        'wifi': wifi_ds,\n                        'ibeacon': np.zeros((0, 3))\n                    }\n\n        if ibeacon_datas.size != 0:\n            sep_tss = np.unique(ibeacon_datas[:, 0].astype(float))\n            ibeacon_datas_list = split_ts_seq(ibeacon_datas, sep_tss)\n            for ibeacon_ds in ibeacon_datas_list:\n                diff = np.abs(step_positions[:, 0] - float(ibeacon_ds[0, 0]))\n                index = np.argmin(diff)\n                target_xy_key = tuple(step_positions[index, 1:3])\n                if target_xy_key in mwi_datas:\n                    mwi_datas[target_xy_key]['ibeacon'] = np.append(mwi_datas[target_xy_key]['ibeacon'], ibeacon_ds, axis=0)\n                else:\n                    mwi_datas[target_xy_key] = {\n                        'magnetic': np.zeros((0, 4)),\n                        'wifi': np.zeros((0, 5)),\n                        'ibeacon': ibeacon_ds\n                    }\n\n        sep_tss = np.unique(magn_datas[:, 0].astype(float))\n        magn_datas_list = split_ts_seq(magn_datas, sep_tss)\n        for magn_ds in magn_datas_list:\n            diff = np.abs(step_positions[:, 0] - float(magn_ds[0, 0]))\n            index = np.argmin(diff)\n            target_xy_key = tuple(step_positions[index, 1:3])\n            if target_xy_key in mwi_datas:\n                mwi_datas[target_xy_key]['magnetic'] = np.append(mwi_datas[target_xy_key]['magnetic'], magn_ds, axis=0)\n            else:\n                mwi_datas[target_xy_key] = {\n                    'magnetic': magn_ds,\n                    'wifi': np.zeros((0, 5)),\n                    'ibeacon': np.zeros((0, 3))\n                }\n\n    return mwi_datas\n\n\ndef extract_magnetic_strength(mwi_datas):\n    magnetic_strength = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        magnetic_data = mwi_datas[position_key]['magnetic']\n        magnetic_s = np.mean(np.sqrt(np.sum(magnetic_data[:, 1:4] ** 2, axis=1)))\n        magnetic_strength[position_key] = magnetic_s\n\n    return magnetic_strength\n\n\ndef extract_wifi_rssi(mwi_datas):\n    wifi_rssi = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        wifi_data = mwi_datas[position_key]['wifi']\n        for wifi_d in wifi_data:\n            bssid = wifi_d[2]\n            rssi = int(wifi_d[3])\n\n            if bssid in wifi_rssi:\n                position_rssi = wifi_rssi[bssid]\n                if position_key in position_rssi:\n                    old_rssi = position_rssi[position_key][0]\n                    old_count = position_rssi[position_key][1]\n                    position_rssi[position_key][0] = (old_rssi * old_count + rssi) / (old_count + 1)\n                    position_rssi[position_key][1] = old_count + 1\n                else:\n                    position_rssi[position_key] = np.array([rssi, 1])\n            else:\n                position_rssi = {}\n                position_rssi[position_key] = np.array([rssi, 1])\n\n            wifi_rssi[bssid] = position_rssi\n\n    return wifi_rssi\n\n\ndef extract_ibeacon_rssi(mwi_datas):\n    ibeacon_rssi = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        ibeacon_data = mwi_datas[position_key]['ibeacon']\n        for ibeacon_d in ibeacon_data:\n            ummid = ibeacon_d[1]\n            rssi = int(ibeacon_d[2])\n\n            if ummid in ibeacon_rssi:\n                position_rssi = ibeacon_rssi[ummid]\n                if position_key in position_rssi:\n                    old_rssi = position_rssi[position_key][0]\n                    old_count = position_rssi[position_key][1]\n                    position_rssi[position_key][0] = (old_rssi * old_count + rssi) / (old_count + 1)\n                    position_rssi[position_key][1] = old_count + 1\n                else:\n                    position_rssi[position_key] = np.array([rssi, 1])\n            else:\n                position_rssi = {}\n                position_rssi[position_key] = np.array([rssi, 1])\n\n            ibeacon_rssi[ummid] = position_rssi\n\n    return ibeacon_rssi\n\n\ndef extract_wifi_count(mwi_datas):\n    wifi_counts = {}\n    for position_key in mwi_datas:\n        # print(f'Position: {position_key}')\n\n        wifi_data = mwi_datas[position_key]['wifi']\n        count = np.unique(wifi_data[:, 2]).shape[0]\n        wifi_counts[position_key] = count\n\n    return wifi_counts","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n# <div class=\"alert alert-block alert-warning\">Setting</div> \n\nTo run the sample code, set up the data path and other settings. Also, let's check the directory structure at the same time.\n\n## <u>Note</u>\n\nCells where you can change the analysis target by changing the values, I'll mark with orange sentence <span style=\"color: orange; \">like this</span>!!!"},{"metadata":{},"cell_type":"markdown","source":"First, in order to process the data, we have to specify four values.\n\n- Directory for metadata\n\n- state (id under train and test directories)\n\n- floor (id under state directory)\n\n- Directory for data (train or test)\n\nSince the  values are decisive in original main.py, I define them as a variable."},{"metadata":{},"cell_type":"markdown","source":"### <span style=\"color: orange; \">Set site & floor you want to analyze↓↓↓</span>"},{"metadata":{"trusted":true},"cell_type":"code","source":"#User setting\ndata_root = \"../input/indoor-location-navigation/train\"\nmetadata_root = \"../input/indoor-location-navigation/metadata\"\nstate = \"5a0546857ecc773753327266\"\nfloor = \"B1\"","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def list_files(startpath):\n    \"\"\"Show directory structure recursive like tree command.\n    refered from https://stackoverflow.com/questions/9727673/list-directory-tree-structure-in-python\n    \"\"\"\n    for root, dirs, files in os.walk(startpath):\n        level = root.replace(startpath, '').count(os.sep)\n        indent = ' ' * 4 * (level)\n        print(f'{indent}{os.path.basename(root)}/')\n        subindent = ' ' * 4 * (level + 1)\n        for f in files:\n            print(f'{subindent}{f}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the directory structure so that we can freely set the path we did earlier.\n\nCheck it from the indoor trace data. The root of the data is train or test. Take a look under the train directory. Under this directory is like \"site/floor/data\". Let's try listing under \"5cd56c0ce2acfd2d33b6ab27\" site."},{"metadata":{"trusted":true},"cell_type":"code","source":"#list_files is utility function defined in previous hidden cell.\nlist_files(\"../input/indoor-location-navigation/train/5cd56c0ce2acfd2d33b6ab27\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataset for this competition consists of dense indoor signatures of WiFi, geomagnetic field, iBeacons etc., as well as ground truth (waypoint) collected from hundreds of buildings in Chinese cities. The data found in path trace files (*.txt) corresponds to an indoor path between position p_1 and p_2 walked by a site-surveyor.\n\nIn other words, the data listed above is the path trace data collected from a certain 5cd56c0ce2acfd2d33b6ab27 site in China.\n\nDuring the walk, an Android smartphone is held flat in front of the surveyors body, and a sensor data recording app is running on the device to collect IMU (accelerometer, gyroscope) and geomagnetic field (magnetometer) readings, as well as WiFi and Bluetooth iBeacon scanning results.\n\nLet's check a path trace data file."},{"metadata":{"trusted":true},"cell_type":"code","source":"!head ../input/indoor-location-navigation/train/5cd56c0ce2acfd2d33b6ab27/B1/5d09a625bd54340008acddb9.txt -n 15","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The lines with '#' at the beginning are metadata. It contains the location where the measurement was taken, the Android smartphone model and OS version, etc.\n\nRows without '#' are the measured data. The first column is the Unix Time in millisecond. The second column is the type of data.　For details on the second row, [Android Developers Document](https://developer.android.com/guide/topics/sensors/sensors_overview) is referable. In particular, TYPE_WAYPOINT is our target and is not included in the test data. ~~There is only one TYPE_WAYPOINT listed per .txt file.~~ Some trace path files seem to containt some TYPE_WAYPOINT line of defferent timestamp.\n\nThe third and subsequent columns are the measured values. See [here](https://github.com/tasotasoso/indoor-location-competition-20#sample-data) for details."},{"metadata":{},"cell_type":"markdown","source":"The same goes for metadata.\n\nAs metadata, there are following three data for each floor.\n- floor_image.png\n- floor_info.json\n- geojson_map.json"},{"metadata":{"trusted":true},"cell_type":"code","source":"list_files(metadata_root + \"/5cd56c0ce2acfd2d33b6ab27\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#The above settings should allow the data to be loaded.\nfloor_data_dir = '/'.join([metadata_root, state, floor])\npath_data_dir = '/'.join([data_root, state, floor])\nfloor_plan_filename = '/'.join([floor_data_dir, 'floor_image.png'])\nfloor_info_filename = '/'.join([floor_data_dir, 'floor_info.json'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, I'll set the processing result will be output as follows."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Output setting\nsave_dir = '/'.join(['./output', state, floor])\npath_image_save_dir = '/'.join([save_dir, 'path_images'])\nstep_position_image_save_dir = save_dir\nmagn_image_save_dir = save_dir\nwifi_image_save_dir = '/'.join([save_dir, 'wifi_images'])\nibeacon_image_save_dir = '/'.join([save_dir, 'ibeacon_images'])\nwifi_count_image_save_dir = save_dir","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#If the directories do not exist, they should be created automatically.\nPath(path_image_save_dir).mkdir(parents=True, exist_ok=True)\nPath(magn_image_save_dir).mkdir(parents=True, exist_ok=True)\nPath(wifi_image_save_dir).mkdir(parents=True, exist_ok=True)\nPath(ibeacon_image_save_dir).mkdir(parents=True, exist_ok=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n# <div class=\"alert alert-block alert-warning\">Run the tool</div> \n\nOK, we're ready to go. Let's proceed with the process corresponding to main.py. The code is somewhat long, so I'll let some cells hide . Please display and check them if necessary."},{"metadata":{},"cell_type":"markdown","source":"## Create ground truth location, step detection and geo-magnetic field intensity visualization"},{"metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"with open(floor_info_filename) as f:\n    floor_info = json.load(f)\nwidth_meter = floor_info[\"map_info\"][\"width\"]\nheight_meter = floor_info[\"map_info\"][\"height\"]\npath_filenames = list(Path(path_data_dir).resolve().glob(\"*.txt\"))\n\n# 1. visualize ground truth positions\nprint('Visualizing ground truth positions...')\nfor path_filename in path_filenames:\n    print(f'Processing file: {path_filename}...')\n    path_data = read_data_file(path_filename)\n    path_id = path_filename.name.split(\".\")[0]\n    fig = visualize_trajectory(path_data.waypoint[:, 1:3], floor_plan_filename, width_meter, height_meter, title=path_id, show=False)\n    html_filename = f'{path_image_save_dir}/{path_id}.html'\n    html_filename = str(Path(html_filename).resolve())\n    save_figure_to_html(fig, html_filename)\n    \n# 2. visualize step position, magnetic, wifi, ibeacon\nprint('Visualizing more information...')\nmwi_datas = calibrate_magnetic_wifi_ibeacon_to_position(path_filenames)\nstep_positions = np.array(list(mwi_datas.keys()))\nfig = visualize_trajectory(step_positions, floor_plan_filename, width_meter, height_meter, mode='markers', title='Step Position', show=True)\nhtml_filename = f'{step_position_image_save_dir}/step_position.html'\nhtml_filename = str(Path(html_filename).resolve())\nsave_figure_to_html(fig, html_filename)\nmagnetic_strength = extract_magnetic_strength(mwi_datas)\nheat_positions = np.array(list(magnetic_strength.keys()))\nheat_values = np.array(list(magnetic_strength.values()))\nfig = visualize_heatmap(heat_positions, heat_values, floor_plan_filename, width_meter, height_meter, colorbar_title='mu tesla', title='Magnetic Strength', show=True)\nhtml_filename = f'{magn_image_save_dir}/magnetic_strength.html'\nhtml_filename = str(Path(html_filename).resolve())\nsave_figure_to_html(fig, html_filename)\nwifi_rssi = extract_wifi_rssi(mwi_datas)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create Wifi RSSI heatmap"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'This floor has {len(wifi_rssi.keys())} wifi aps')\nten_wifi_bssids = list(wifi_rssi.keys())[0:10]\nprint('Example 10 wifi ap bssids:\\n')\nfor bssid in ten_wifi_bssids:\n    print(bssid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <span style=\"color: orange; \">We can choose wifi_bssids↓↓↓</span>"},{"metadata":{"trusted":true},"cell_type":"code","source":"target_wifi = \"db01605eac3f33540038bd9722aba25774871d43\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"heat_positions = np.array(list(wifi_rssi[target_wifi].keys()))\nheat_values = np.array(list(wifi_rssi[target_wifi].values()))[:, 0]\nfig = visualize_heatmap(heat_positions, heat_values, floor_plan_filename, width_meter, height_meter, colorbar_title='dBm', title=f'Wifi: {target_wifi} RSSI', show=False)\nhtml_filename = f'{wifi_image_save_dir}/{target_wifi.replace(\":\", \"-\")}.html'\nhtml_filename = str(Path(html_filename).resolve())\nsave_figure_to_html(fig, html_filename)\nibeacon_rssi = extract_ibeacon_rssi(mwi_datas)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Create iBeacon RSSI heatmap and Wifi Count heatmap"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'This floor has {len(ibeacon_rssi.keys())} ibeacons')\nten_ibeacon_ummids = list(ibeacon_rssi.keys())[0:10]\nprint('Example 10 ibeacon UUID_MajorID_MinorIDs:\\n')\nfor ummid in ten_ibeacon_ummids:\n    print(ummid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <span style=\"color: orange; \">We can choose ibeacon_ummid↓↓↓</span>"},{"metadata":{"trusted":true},"cell_type":"code","source":"target_ibeacon = \"89cb11b04122cef23388b0da06bd426c1f48a9b5_cfc84f0752adc96b489f71195d91a946c5f6d3e8_8159618423dfa22f1ca0b62543e2f18eef630ce8\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"heat_positions = np.array(list(ibeacon_rssi[target_ibeacon].keys()))\nheat_values = np.array(list(ibeacon_rssi[target_ibeacon].values()))[:, 0]\nfig = visualize_heatmap(heat_positions, heat_values, floor_plan_filename, width_meter, height_meter, colorbar_title='dBm', title=f'iBeacon: {target_ibeacon} RSSI', show=False)\nhtml_filename = f'{ibeacon_image_save_dir}/{target_ibeacon}.html'\nhtml_filename = str(Path(html_filename).resolve())\nsave_figure_to_html(fig, html_filename)\nwifi_counts = extract_wifi_count(mwi_datas)\nheat_positions = np.array(list(wifi_counts.keys()))\nheat_values = np.array(list(wifi_counts.values()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# filter out positions that no wifi detected\nmask = heat_values != 0\nheat_positions = heat_positions[mask]\nheat_values = heat_values[mask]\nfig = visualize_heatmap(heat_positions, heat_values, floor_plan_filename, width_meter, height_meter, colorbar_title='number', title=f'Wifi Count', show=False)\nhtml_filename = f'{wifi_count_image_save_dir}/wifi_count.html'\nhtml_filename = str(Path(html_filename).resolve())\nsave_figure_to_html(fig, html_filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n# <div class=\"alert alert-block alert-success\">Check output data</div>\n\nWe got some processed output data.\n\n- path_images: Ground truth location visualization\n\n- step_position.html: Sample step detection and visualization\n\n- magnetic_strength.html: Geo-magnetic field intensity visualization\n\n- wifi_images: WiFi RSSI heatmap generation\n\n- ibeacon_images: iBeacon RSSI heatmap generation\n\n- wifi_count.html: WiFi SSID counts visualization"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ./output/5a0546857ecc773753327266/B1/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see each outputs."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!ls ./output/5a0546857ecc773753327266/B1/path_images\t    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IFrame(src='./output/5a0546857ecc773753327266/B1/path_images/5e158ee11506f2000638fd0f.html', width=950, height=950)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IFrame(src='./output/5a0546857ecc773753327266/B1/step_position.html', width=950, height=950)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IFrame(src='./output/5a0546857ecc773753327266/B1/magnetic_strength.html', width=950, height=950)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ./output/5a0546857ecc773753327266/B1/wifi_images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IFrame(src='./output/5a0546857ecc773753327266/B1/wifi_images/db01605eac3f33540038bd9722aba25774871d43.html', width=950, height=950)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ./output/5a0546857ecc773753327266/B1/ibeacon_images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IFrame(src='./output/5a0546857ecc773753327266/B1/ibeacon_images/89cb11b04122cef23388b0da06bd426c1f48a9b5_cfc84f0752adc96b489f71195d91a946c5f6d3e8_8159618423dfa22f1ca0b62543e2f18eef630ce8.html', width=950, height=950)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IFrame(src='./output/5a0546857ecc773753327266/B1/wifi_count.html', width=950, height=950)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a> <br>\n# <div class=\"alert alert-block alert-success\">How script extracts data from trace path file?</div>\n\nIt very helpful to preprocess data, so I will describe it here. This script uses the read_data_file function of the io_f.py module to parse trace path files."},{"metadata":{"trusted":true},"cell_type":"code","source":"from io_f import read_data_file","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This module can read given path trace file, and return dataclass instance which has following menbers. By accessing these members, we can get the data of each path trace file in numpy array format."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\n@dataclass\nclass ReadData:\n    acce: np.ndarray\n    acce_uncali: np.ndarray\n    gyro: np.ndarray\n    gyro_uncali: np.ndarray\n    magn: np.ndarray\n    magn_uncali: np.ndarray\n    ahrs: np.ndarray\n    wifi: np.ndarray\n    ibeacon: np.ndarray\n    waypoint: np.ndarray\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's analize one path trace file."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_path_trace_file_path = \"../input/indoor-location-navigation/train/5a0546857ecc773753327266/B1/5e15730aa280850006f3d005.txt\"\ndata = read_data_file(sample_path_trace_file_path)\ndata","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can access extracted data like this."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.acce","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Script reads files and parse them with \\t, like following."},{"metadata":{"trusted":true},"cell_type":"code","source":"#From https://github.com/tasotasoso/indoor-location-competition-20/blob/master/io_f.py\n\n\"\"\"\n        line_data = line_data.split('\\t')\n\n        if line_data[1] == 'TYPE_ACCELEROMETER':\n            acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n            acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE':\n            gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n            gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n            magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n            magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ROTATION_VECTOR':\n            ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_WIFI':\n            sys_ts = line_data[0]\n            ssid = line_data[2]\n            bssid = line_data[3]\n            rssi = line_data[4]\n            lastseen_ts = line_data[6]\n            wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n            wifi.append(wifi_data)\n            continue\n\n        if line_data[1] == 'TYPE_BEACON':\n            ts = line_data[0]\n            uuid = line_data[2]\n            major = line_data[3]\n            minor = line_data[4]\n            rssi = line_data[6]\n            ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n            ibeacon.append(ibeacon_data)\n            continue\n\n        if line_data[1] == 'TYPE_WAYPOINT':\n            waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For example, we can extract all data types and turn them into a pandas data frame like following way."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_acce = pd.DataFrame()\ntrain_acce_uncali = pd.DataFrame()\ntrain_gyro = pd.DataFrame()\ntrain_gyro_uncali = pd.DataFrame()\ntrain_magn = pd.DataFrame()\ntrain_magn_uncali = pd.DataFrame()\ntrain_ahrs = pd.DataFrame()\ntrain_wifi = pd.DataFrame()\ntrain_ibeacon = pd.DataFrame()\ntrain_waypoint = pd.DataFrame()\n\ndfs_train = {\n    \"acce\": train_acce,\n    \"acce_uncali\": train_acce_uncali,\n    \"gyro\": train_gyro,\n    \"gyro_uncali\": train_gyro_uncali,\n    \"magn\": train_magn,\n    \"magn_uncali\": train_magn_uncali,\n    \"ahrs\": train_ahrs,\n    \"wifi\": train_wifi,\n    \"ibeacon\": train_ibeacon,\n    \"waypoint\": train_waypoint \n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def concat_data(dfs, file_path):\n    \"\"\"Extract data from trace path file and concat them to given pandas dataframe.\n    \"\"\"\n    data = read_data_file(file_path)\n    file_path_parsed =  file_path.split(\"/\")\n    floor = file_path_parsed[-2]\n    trace_file = file_path_parsed[-1].split(\".\")[0]    \n\n    if data.acce.size > 0:        \n        df_tmp = pd.DataFrame(data.acce, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"acce\"] = pd.concat([dfs[\"acce\"], df_tmp])\n        \n    if data.acce_uncali.size > 0:                              \n        df_tmp = pd.DataFrame(data.acce_uncali, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"acce_uncali\"] = pd.concat([dfs[\"acce_uncali\"], df_tmp])\n    \n    if data.gyro.size > 0:                              \n        df_tmp =  pd.DataFrame(data.gyro, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"gyro\"] = pd.concat([dfs[\"gyro\"], df_tmp])\n    \n    if data.gyro_uncali.size > 0:                              \n        df_tmp = pd.DataFrame(data.gyro_uncali, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"gyro_uncali\"] = pd.concat([dfs[\"gyro_uncali\"], df_tmp])\n    \n    if data.magn.size > 0:                              \n        df_tmp = pd.DataFrame(data.magn, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"magn\"] = pd.concat([dfs[\"magn\"], df_tmp])\n\n    if data.magn_uncali.size > 0:    \n        df_tmp = pd.DataFrame(data.magn_uncali, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"magn_uncali\"] = pd.concat([dfs[\"magn_uncali\"], df_tmp])\n\n    if data.ahrs.size > 0:    \n        df_tmp = pd.DataFrame(data.ahrs, columns=(\"time_stamp\", \"x\", \"y\", \"z\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"ahrs\"] = pd.concat([dfs[\"ahrs\"], df_tmp])\n    \n    if data.wifi.size > 0:    \n        df_tmp = pd.DataFrame(data.wifi, columns=(\"time_stamp\", \"ssid\", \"bssid\", \"rssi\",\"lastseen_ts\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"wifi\"] = pd.concat([dfs[\"wifi\"], df_tmp])\n \n    if data.ibeacon.size > 0:    \n        df_tmp = pd.DataFrame(data.ibeacon, columns=(\"time_stamp\", \"uuid_major_minor\", \"rssi\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"ibeacon\"] = pd.concat([dfs[\"ibeacon\"], df_tmp])\n\n    if data.waypoint.size > 0:    \n        df_tmp = pd.DataFrame(data.waypoint , columns=(\"time_stamp\", \"x\", \"y\"))\n        df_tmp[\"floor\"] = floor\n        df_tmp[\"trace_file\"] = trace_file\n        dfs[\"waypoint\"] = pd.concat([dfs[\"waypoint\"], df_tmp])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_data(dfs_train, sample_path_trace_file_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dfs_train[\"acce\"]","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}