{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Introduction**\n\nYou probably have experienced this before: You are in an underground carpark and you have just activated your navigation system. But the navigation system has trouble locating you on the map due to poor GPS signal quality caused by the concrete walls. Although, in this use case a better accuracy would only be a \"nice to have\", in other cases it could become a necessity for indoor applications.\n\nTo improve the accuracy of indoor positioning systems, we are asked to predict the indoor position of smartphones based on real-time sensor data in this competition.\n","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom dataclasses import dataclass\n\nimport matplotlib.pyplot as plt # visualization\nplt.rcParams.update({'font.size': 14})\nimport seaborn as sns # visualization\n\nimport warnings # Supress warnings \nwarnings.filterwarnings('ignore')\n\nfrom tqdm import tqdm\n\nimport json\nimport plotly.graph_objs as go\nfrom PIL import Image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Dataset Overview**\n\nThe dataset we are working with is provided by the Chinese company XYZ10 specialized in indoor positioning technology. The dataset consists of path trace recordings of a person walking from one point to another. During the walk, the following sensor signals are recorded:\n\n* accelerometer\n* magnetic field\n* gyroscope\n* rotation vector\n* WiFi\n* Bluetooth iBeacon\n* ground truths (waypoint locations)\n\n\nLet's have a first look at one of the trace files to get a rough feeling for the data. Unfortunately, this time we don't have the comfort of .csv format. Instead, we are provided text files.","metadata":{}},{"cell_type":"markdown","source":"# **Unix Timestamp**\n\nThe first column is the Unix Time in milliseconds. If you are not familiar with Unix time, then I recommend reading up on it on wikipedia. But in short, the unix time is the time elapsed since 00:00:00 UTC on 1 January 1970.\n\nAt this point, I am not yet sure if we really need to convert Unix timestamps to human understandable timestamps but here is the conversion - just in case. Since we are working with milliseconds, we need to divide the timestamps by 1000. The below sample starts at 1573713056850 and ends at 1573713091483, which corresponds to a short 34.633 s long trace done on November 14th 2019.","metadata":{}},{"cell_type":"code","source":"from datetime import datetime\nstart_time = 1573713056850\nend_time = 1573713091483\n\nprint(datetime.fromtimestamp(start_time/1000.0))\nprint(datetime.fromtimestamp(end_time/1000.0))\nprint(datetime.fromtimestamp(end_time/1000.0)-datetime.fromtimestamp(start_time/1000.0))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Waypoint**\n\nA waypoint is a point of reference that can be used for location and navigation. Waypoints can be the specific latitude and longitude of a location, a well-known building or natural feature.Let's plot the trace of the waypoint on the map first to get a feeling for this example.\n\nThe following code is also copied and edited from <a href=\"https://www.kaggle.com/ihelon/indoor-location-exploratory-data-analysis\">@ihelon's notebook</a> and is originally from the <a href=\"https://github.com/location-competition/indoor-location-competition-20/blob/master/visualize_f.py\">competition's Github page.</a>\n","metadata":{}},{"cell_type":"code","source":"\n@dataclass\nclass ReadData:\n    acce: np.ndarray\n    acce_uncali: np.ndarray\n    gyro: np.ndarray\n    gyro_uncali: np.ndarray\n    magn: np.ndarray\n    magn_uncali: np.ndarray\n    ahrs: np.ndarray\n    wifi: np.ndarray\n    ibeacon: np.ndarray\n    waypoint: np.ndarray\n\n\ndef read_data_file(data_filename):\n    acce = []\n    acce_uncali = []\n    gyro = []\n    gyro_uncali = []\n    magn = []\n    magn_uncali = []\n    ahrs = []\n    wifi = []\n    ibeacon = []\n    waypoint = []\n    \n    \n    with open(data_filename, 'r', encoding='utf-8') as file:\n        lines = file.readlines()\n\n    for line_data in lines:\n        line_data = line_data.strip()\n        if not line_data or line_data[0] == '#':\n            continue\n\n        line_data = line_data.split('\\t')\n\n        if line_data[1] == 'TYPE_WAYPOINT':\n            waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n            continue\n            \n        if line_data[1] == 'TYPE_ACCELEROMETER':\n            acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n            acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_GYROSCOPE':\n            gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n            gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n            magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n            magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ROTATION_VECTOR':\n            ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_WIFI':\n            sys_ts = line_data[0]\n            ssid = line_data[2]\n            bssid = line_data[3]\n            rssi = line_data[4]\n            lastseen_ts = line_data[6]\n            wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n            wifi.append(wifi_data)\n            continue\n\n        if line_data[1] == 'TYPE_BEACON':\n            ts = line_data[0]\n            uuid = line_data[2]\n            major = line_data[3]\n            minor = line_data[4]\n            rssi = line_data[6]\n            ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n            ibeacon.append(ibeacon_data)\n            continue\n            \n    acce = np.array(acce)\n    acce_uncali = np.array(acce_uncali)\n    gyro = np.array(gyro)\n    gyro_uncali = np.array(gyro_uncali)\n    magn = np.array(magn)\n    magn_uncali = np.array(magn_uncali)\n    ahrs = np.array(ahrs)\n    wifi = np.array(wifi)\n    ibeacon = np.array(ibeacon)\n    waypoint = np.array(waypoint)\n    \n    return ReadData(acce, acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs, wifi, ibeacon, waypoint)\n    \nsample_file = read_data_file(\"../input/indoor-location-navigation/train/5a0546857ecc773753327266/F2/5dccf516c04f060006e6e3c9.txt\")\n\nprint('acce shape:', sample_file.acce.shape)\nprint('acce_uncali shape:', sample_file.acce_uncali.shape)\nprint('gyro shape:', sample_file.gyro.shape)\nprint('gyro_uncali shape:', sample_file.gyro_uncali.shape)\nprint('magn shape:', sample_file.magn.shape)\nprint('magn_uncali shape:',sample_file.magn_uncali.shape)\nprint('ahrs shape:', sample_file.ahrs.shape)\nprint('wifi shape:', sample_file.wifi.shape)\nprint('ibeacon shape:', sample_file.ibeacon.shape)\nprint('waypoint shape:', sample_file.waypoint.shape)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"waypoint_df = pd.DataFrame(sample_file.waypoint)\nwaypoint_df.columns = ['timestamp', 'waypoint_x','waypoint_y']\ndisplay(waypoint_df.style.set_caption('Waypoint'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Inertial Measurement Unit (IMU)**\n\nThe inertial measurement unit (IMU) is a sensor that measures the force, angular rate and orientation of a body. In this case, the body is a phone. These values are measured by accelerometers, gyroscopes, and in this case also magnetometers.\n\n* Accelerometer: Measures change in velocity  \n* Gyroscopes: Measures change in rotation  \n* Magnetometer: Measures magnetic field \n\nThe IMU sensor data has the same shape in this case. Note, that this is true for a lot of traces but not all of them. We can concatenate them to a dataframe for the initial analysis of the data.\n\n","metadata":{}},{"cell_type":"markdown","source":"# **Analysing The Data**","metadata":{}},{"cell_type":"code","source":"import os\nimport glob\nfrom PIL import Image\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport json\nfrom tqdm.notebook import tqdm\nfrom pathlib import Path\nfrom dataclasses import dataclass\ndataset_path = Path('../input/indoor-location-navigation')\nos.listdir(dataset_path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sites = os.listdir(dataset_path/\"train\")\nprint(f'There are {len(train_sites)} sites in the training set')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"example_site = os.listdir(dataset_path/\"train\")[10]\nexample_site_path = dataset_path/\"train\"/example_site\nprint('Floors for example site:')\nprint(os.listdir(example_site_path))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"floors_per_site = []\nfor i in os.listdir(dataset_path/\"train\"): floors_per_site.append(len(os.listdir(dataset_path/\"train\"/i)))\nprint(f'There are a total of {sum(floors_per_site)} floors. On average, each site has {np.mean(floors_per_site)} floors')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Path text files for example floor:')\nprint(os.listdir(example_site_path/'B1'))\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"There are {len(list((dataset_path/'train').rglob('*.txt')))} path text files in the training set\")\nprint(f\"There are {len(os.listdir(dataset_path/'test'))} path text files in the test set\")\nprint(f'There are {len(os.listdir(dataset_path/\"metadata\"))} sites in the metadata, just like the training set')\n\nmetadata_example_site = os.listdir(dataset_path/\"metadata\")[10]\nmetadata_example_site_path = dataset_path/\"metadata\"/metadata_example_site\nmetadata_example_floor_path = dataset_path/\"metadata\"/metadata_example_site/os.listdir(metadata_example_site_path)[0]\nprint(os.listdir(metadata_example_floor_path))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Image.open(metadata_example_floor_path/'floor_image.png')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(metadata_example_floor_path/'geojson_map.json') as geojson_map:\n    data = json.load(geojson_map)\n    geojson_map.close()\nprint(data)\n\nwith open(metadata_example_floor_path/'floor_info.json') as floor_info:\n    data = json.load(floor_info)\n    floor_info.close()\nprint(data)\n","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"example_floor_path = example_site_path/'B1'\nexample_txt_path = example_floor_path/os.listdir(example_floor_path)[0]\nwith open(example_txt_path) as example_txt:\n    data = example_txt.read()\n    example_txt.close()\nprint(data)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}