{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n\n## Indoor Location and Navigation\n\nYour smartphone goes everywhere with you—whether driving to the grocery store or shopping for holiday gifts. With your permission, apps can use your location to provide contextual information. You might get driving directions, find a store, or receive alerts for nearby promotions. These handy features are enabled by GPS, which requires outdoor exposure for the best accuracy. Yet, there are many times when you’re inside large structures, such as a shopping mall or event center. Accurate indoor positioning, based on public sensors and user permission, allows for a great location-based experience even when you aren’t outside.\n\nCurrent positioning solutions have poor accuracy, particularly in multi-level buildings, or generalize poorly to small datasets. Additionally, GPS was built for a time before smartphones. Today’s use cases often require more granularity than is typically available indoors.\n\nIn this competition, your task is to predict the indoor position of smartphones based on real-time sensor data, provided by indoor positioning technology company XYZ10 in partnership with Microsoft Research. You'll locate devices using “active” localization data, which is made available with the cooperation of the user. Unlike passive localization methods (e.g. radar, camera), the data provided for this competition requires explicit user permission. You'll work with a dataset of nearly 30,000 traces from over 200 buildings.\n\nIf successful, you’ll contribute to research with broad-reaching possibilities, including industries like manufacturing, retail, and autonomous devices. With more accurate positioning, existing location-based apps could even be improved. Perhaps you’ll even see the benefits yourself the next time you hit the mall."},{"metadata":{},"cell_type":"markdown","source":"\n\n### Notebook Contents\n\nIn this notebook I'll do some Exploratory Data Analysis, trying to update it with new sections in the next weeks.\n\nThe sections are: \n\n0. [**File Structure explained**](#files)
\n 0.1. [*Visual Explanation of our data*](#visual)\n\n1. [**metadata**](#meta_head)
\n \n 1.1. [*Intro*](#meta_intro)
\n \n 1.2. [*Geospatial Intro*](#meta_geo)
\n \n2. [**train**](#train_head)
\n\n 2.1. [*Basic Statistics for Sites, Floors, Paths*](#train_stats)
\n \n 2.2 [*Data explanation and relationships*](#train_expl)
\n \n3. [**test and submission**](#test_and_sub_head)
\n\n\n##### Props to: \n\n[Leonie](https://www.kaggle.com/iamleonie/intro-to-indoor-location-navigation), always a lot to learn from her, [indoor location competition Git](https://github.com/location-competition/indoor-location-competition-20/blob/master/io_f.py), [flaticon](https://www.flaticon.com/) and [imgur](https://imgur.com/) for visualizations.\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\npd.options.display.max_columns = 50\npd.options.display.max_colwidth = 200\nimport os\n\nfrom dataclasses import dataclass\nimport colorama\nfrom colorama import Fore, Back, Style\nimport folium\nimport json\nimport geopandas as gpd\n\nimport re\nimport pyproj\nfrom pyproj import Proj, transform\n\nfrom shapely.ops import cascaded_union\nimport matplotlib\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'figure.max_open_warning': 0})\nplt.style.use('fivethirtyeight')\nimport seaborn as sns # visualization\nimport warnings # Supress warnings \nwarnings.filterwarnings('ignore')\n\nimport plotly.graph_objs as go\nfrom PIL import Image\n\nfrom tqdm import tqdm\n\nmetadata_path = '/kaggle/input/indoor-location-navigation/metadata/'\ntrain_path = '/kaggle/input/indoor-location-navigation/train/'\ntest_path = '/kaggle/input/indoor-location-navigation/test/'\n\ny_ = Fore.YELLOW\nr_ = Fore.RED\ng_ = Fore.GREEN\nb_ = Fore.BLUE\nm_ = Fore.MAGENTA\nc_ = Fore.CYAN\nsr_ = Style.RESET_ALL\n\ncolor_dict = {'site': c_, 'floor': y_, 'path': b_}\n\ntest_structure = {test_path: ['path_1.txt','path_2.txt','path_3.txt','...', 'path_n.txt']}\n\nmetadata_structure = {metadata_path: \n {'site_1': {'floor_1': ['geojson_map.json', 'floor_info.json', 'floor_image.png'],\n 'floor_2': ['geojson_map.json', 'floor_info.json', 'floor_image.png']},\n 'site_2': {'basement': ['geojson_map.json', 'floor_info.json', 'floor_image.png'],\n 'floor_1': ['geojson_map.json', 'floor_info.json', 'floor_image.png']},\n }\n }\n\ntrain_structure = {train_path: \n {'site_1': {'floor_1': ['path_1.txt', 'path_2.txt'],\n 'floor_2': ['path_1.txt', 'path_2.txt', 'path_3.txt']},\n 'site_2': {'basement': ['path_1.txt'],\n 'floor_1': ['path_1.txt', 'path_2.txt']},\n }\n }\n\ndef pretty(d, indent=0, max_enum = 10):\n for enum, (key, value) in enumerate(d.items()):\n if enum < max_enum:\n if ((len(str(key)) < 5) or (any(x in str(key) for x in ['floor', 'basement']))) and ('site' not in str(key)):\n print('\\t'*indent, color_dict['floor'] + str(key)) \n \n elif ((len(str(key)) > 5)):\n print('\\t'*indent, color_dict['site'] + str(key)) \n \n else:\n print('\\t' * indent + str(key))\n if isinstance(value, dict):\n pretty(value, indent+1)\n else:\n if (len(value)>0) & (any(x in str(value) for x in ['.json', '.txt', '.png'])):\n print(\"\"\"{0}{1}{2}\"\"\".format('\\t'*(indent+1), color_dict['path'], str(value)))\n else: \n print('\\t' * (indent+1) + str(value))\n print(Style.RESET_ALL)\n \ndef create_dict(metadata_path, max_enum = 1000, files_enum = None):\n \n metadata_dict = {}\n sites = os.listdir(metadata_path)\n metadata_dict[metadata_path] = sites\n sites_path = list(map(lambda x: os.path.join(metadata_path, x), sites))\n sites_dict = {}\n for sites_enum, site_path in enumerate(sites_path):\n \n if sites_enum\n

Files Structure

\n\nFiles\ntrain - training path files, organized by site and floor; each path file contains the data of a single path on a single floor\ntest - test path files, organized by site and floor; each path file contains the data of a single path on a single floor, but without the waypoint (x, y) data; the task of this competition is, for a given site-path file, predict the floor and waypoint locations at the timestamps given in the sample_submission.csv file\n\n- **train**: training path files, organized by site and floor; each path files contains the data of a **single** **path** on a **single** **floor**.
\n\n- **test**: test path files, organized by site and floor; each path files contains the data of a single path on a single floor, but **without the waypoint (x, y) data**; the task of this competition is, for a given site-path file, predict the floor and waypoint locations at the timestamps given in the sample_submission.csv file.
\n\n- **metadata**: floor metadata folder, organized by site and floor, which includes the following for each floor:
\n\n
    \n
  1. floor_image.png
    2. \n
  2. floor_info.json
    3. \n
  3. geojson_map.json
    4. \n
\n\n- **sample_submission.csv**: a sample submission file in the correct format; each has a unique id which contains a site id, a path id, and the timestamp within the trace for which to make a prediction; see the Evaluation page for the required integer mapping of floor names
\n\n\nI'll start by visually explaining the relationships between our data. "},{"metadata":{},"cell_type":"markdown","source":"\n#### Visual Explanation of our Data"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"\n### metadata"},{"metadata":{},"cell_type":"markdown","source":"\n#### Intro\n\nFor each site-floor we have 3 files: \n\n"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"pretty(metadata_structure)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see an example for site `5cd56c0ce2acfd2d33b6ab27`"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"site_name_ = '5cd56c0ce2acfd2d33b6ab27'\nsite_path = os.path.join(metadata_path, site_name_)\nsite_structure = {site_path: {'B1': ['geojson_map.json', 'floor_info.json', 'floor_image.png'],\n 'F3': ['geojson_map.json', 'floor_info.json', 'floor_image.png'],\n 'F2': ['geojson_map.json', 'floor_info.json', 'floor_image.png']}}\npretty(site_structure)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And floor `B1`"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"floor_info = pd.read_json(os.path.join(site_path, 'B1/floor_info.json'))\nfloor_image = plt.imread(os.path.join(site_path, 'B1/floor_image.png'))\nfloor_geo = (gpd.GeoDataFrame.from_features(\n pd.read_json(os.path.join(site_path, 'B1/geojson_map.json'))['features'])\n .assign(site_name=site_name_))\nprint('Floor Info')\ndisplay(floor_info)\n\nfig, axes = plt.subplots(1, 2, figsize = (16, 10))\nax = axes.ravel()\nfloor_geo['geometry'].plot(ax=ax[0], color = 'red')\nax[0].set_title('Floor {} polygon'.format('B1'))\nax[1].imshow(floor_image)\nax[1].set_title('Floor {} image'.format('B1'))\nfig.suptitle('Floor Polygon and corresponding Floor Image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For each site-floor there's a MultiPolygon, which is the same for all floors in that sites: in fact, it is just the convex hull of the union of each floor polygons. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 3, figsize = (20, 12))\nax = axes.ravel()\n\nsingle_poly_df = (floor_geo.loc[floor_geo.geometry.apply(lambda x: x.geom_type == 'Polygon')]\n .reset_index(drop = True))\n\nfor j in range(len(single_poly_df)):\n single_poly_df.iloc[[j]].plot(ax = ax[j])\n ax[j].set_title(\"Polygon {}\".format(j+1))\n \npolygons = []\nboundary = gpd.GeoSeries(cascaded_union(single_poly_df.geometry.tolist()))\nboundary.plot(color = 'red', ax = ax[4])\nax[4].set_title('Polygon Unions')\nfloor_geo.iloc[[0]]['geometry'].plot(ax = ax[5], color = 'orange')\nax[5].set_title('MultiPolygon')\nplt.suptitle('Floor {} at Site {} Polygons'.format('B1', site_name_ ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### Geospatial Intro\n#### Where are our Polygons in the World?\n\nHere I'll try to show where are the polygons (i.e. parking lots, malls, airports) we are dealing with in this challenge. "},{"metadata":{},"cell_type":"markdown","source":"Here I load all the sites geojson information:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"geo_dfs = []\ngeo_cols = [\"geometry\",\"Vr\",\"category\",\"name\",\"code\",\"floor_num\", 'sid',\n \"type\",\"id\",\"version\",\"display\",\"point\",\"points\",\"doors\", \"site_name\"]\n\nproblematic_sites = []\nfor site in os.listdir(metadata_path):\n site_path = os.path.join(metadata_path, site)\n for floor in os.listdir(site_path):\n floor_path = os.path.join(site_path, floor)\n try:\n geo_df = (gpd.GeoDataFrame.from_features(\n pd.read_json(os.path.join(floor_path, 'geojson_map.json'))['features'])\n .assign(site_name=site))\n except:\n problematic_sites+=[site]\n geo_dfs.append(geo_df)\nproblematic_sites=list(set(problematic_sites))\nfull_geo_df = pd.concat(geo_dfs, axis = 0, ignore_index = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"full_geo_df[['geometry', 'point', 'site_name']].sample()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Points are in **epsg:3857** coordinates, as you can check [here](https://xserver2-dashboard.cloud.ptvgroup.com/dashboard/Content/TechnicalConcepts/Basics/DSC_About_CoordinateSystems.htm). \n\nLet's convert to standard *Latitude* and *Longitude* and plot them on a map. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_lat_lon(point, proj = pyproj.Transformer.from_crs(3857, 4326, always_xy=True)):\n try:\n x1, y1 = point[0], point[1]\n lon, lat = proj.transform(x1, y1)\n return lat, lon\n except:\n return np.nan\n\ndef get_point(x, i=0):\n try:\n return x[i]\n except:\n return np.nan\n \nfull_geo_df_sample = full_geo_df.sample(500).reset_index(drop = True)\nfull_geo_df_sample['lat_lon'] = full_geo_df_sample.point.apply(get_lat_lon)\nfull_geo_df_sample['lat'] = full_geo_df_sample['lat_lon'].apply(lambda x: get_point(x,0))\nfull_geo_df_sample['lon'] = full_geo_df_sample['lat_lon'].apply(lambda x: get_point(x,1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Map for some of our sites"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"m = folium.Map(location=[30.7444062,121.1146543], tiles='openstreetmap', zoom_start = 8)\n\nfor j in range(len(full_geo_df_sample)):\n try:\n folium.Marker(location=[full_geo_df_sample['lat'][j],\n full_geo_df_sample['lon'][j]],\n popup=full_geo_df_sample['site_name'][j],\n icon = folium.Icon(prefix = 'fa', icon = \"map-pin\", color = 'blue'),\n fill_color='#132b5e', num_sides=3, radius=5).add_to(m)\n except:\n continue\nm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the next few days I'll go on in customizing this analysis."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"\n\n### Train"},{"metadata":{"trusted":true},"cell_type":"code","source":"pretty(train_structure)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n#### Basic Statistics for Sites, Floors, Paths\n\nLet's see the number of paths per floor/site. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_dict = create_dict(train_path)[train_path]\ntrain_path_df = pd.DataFrame.from_dict(train_dict, orient = 'index')\n\nassert train_path_df[train_path_df == 0].sum().sum() == 0, \"Floor present in Site, but no path available\"\n\ntrain_path_df['number_of_floors'] = train_path_df.apply(lambda x: ~x.isna()).sum(axis = 1)\n\ntrain_path_df = (train_path_df.reset_index(drop= False).rename(columns = {'index': 'site'})\n .melt(ignore_index = 'False', id_vars = ['site', 'number_of_floors'], var_name = 'floor',\n value_name = 'number_of_paths'))\n\ntrain_path_df = train_path_df.loc[~train_path_df.number_of_paths.isna()].reset_index(drop = True)\n\ndisplay(train_path_df.sample(3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I Add metadata to retrieve the floor numbers"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"floor_meta_info = (full_geo_df.loc[~full_geo_df.floor_num.isna()]\n [['site_name', 'name', 'floor_num']].reset_index(drop = True))\ntrain_path_df_plus_meta = (train_path_df.merge(floor_meta_info, \n left_on = ['site', 'floor'], right_on = ['site_name', 'name']))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(2, 2, figsize = (20, 12))\nax = axes.ravel()\nplot_df = train_path_df[['site', 'number_of_floors']].drop_duplicates(ignore_index = True)\nax[0]= plt.subplot2grid((2, 2), (0, 0), colspan=1)\nplot_df.number_of_floors.hist(ax = ax[0], bins = 50, color = '#2695f0')\nax[0].set_title('Number of Floors per Site distribution')\n\nax[1]= plt.subplot2grid((2, 2), (0, 1))\ntrain_path_df.number_of_paths.hist(ax = ax[1], bins = 30, color = '#2695f0')\nax[1].set_title('Number of Paths per Floor and Site distribution')\n\nax[2] = plt.subplot2grid((2, 2), (1, 0), colspan=1)\n\nplot_df_3 = (train_path_df_plus_meta.groupby('floor_num').agg({'number_of_paths': ['sum', 'mean']}).reset_index())\nplot_df_3.columns = ['floor_num', 'total_paths', 'avg_paths']\nplot_df_3['avg_paths'] = round(plot_df_3['avg_paths'], 3)\n#plot_df_3 = plot_df_3.melt(id_vars = 'floor_num')\n\nplot_df_3.plot(kind = 'bar', x = 'floor_num', y = 'total_paths', ax = ax[2], color = '#f0b326')\n#plot_df_3.plot(kind = 'bar', x = 'floor_num', y = 'avg_paths', ax = ax[2])\nax[2].set_title('Total Number of Paths per Floor Number')\n\nax[3] = plt.subplot2grid((2, 2), (1, 1), colspan=1)\nplot_df_3.plot(kind = 'bar', x = 'floor_num', y = 'avg_paths', ax = ax[3], color = '#f0b326')\nax[3].set_title('Average Number of Paths per Floor Number')\n\nax[0].set_xlabel('N_floors')\nax[1].set_xlabel('N_paths')\nax[2].set_xlabel('Floor_num')\nax[3].set_xlabel('Floor_num')\n\nax[2].get_legend().remove()\nax[3].get_legend().remove()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n#### Data explanation and relationships\n\nI would suggest to read the official competition [Git README](https://github.com/location-competition/indoor-location-competition-20). I will report the pillar information here. \n\n> Each trace (*.txt) corresponds to an indoor path between position p1 and p2 walked by a site-surveyor. During the walk, site-surveyor is holding an Android smartphone flat in front of his body, and a sensor data recording app is running on the device to collect IMU (accelerometer, gyroscope) and geomagnetic field (magnetometer) readings, as well as WiFi and Bluetooth iBeacon scanning results.\n\nAnd, regarding timestamps:\n\n> In specific, we use SensorEvent.timestamp for sensor data and system time for WiFi and Bluetooth scans.\n\nSo we won't probably find the same timestamps for phone data (accelerometer, gyroscope, magnetic field, ahrs) we have for WiFi/Beacon (Beacon in this case is the same as Bluetooth, for those wondering). \n\n#### Visual Explanation of Train Data\n\n"},{"metadata":{},"cell_type":"markdown","source":"Let's load path data for site `5cd56c0ce2acfd2d33b6ab27`, floor `F2` and path `5d09b22fcfb49b00085466a0`."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"site_floor_path = \"5cd56c0ce2acfd2d33b6ab27/F2/5d09b22fcfb49b00085466a0.txt\"\n\nsample_file = read_data_file(os.path.join(train_path, site_floor_path))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_wifi = pd.DataFrame(sample_file.wifi, columns = ['ts_last_seen', 'wifi_id_1', 'wifi_id_2', 'rssi', 'ts_first_seen'])\nsample_wifi[['ts_first_seen', 'ts_last_seen']] = sample_wifi[['ts_first_seen', 'ts_last_seen']].astype(int)\nsample_beacon = pd.DataFrame(sample_file.ibeacon, columns = ['timestamp', 'beacon_id', 'rssi'])\nsample_beacon['timestamp'] = sample_beacon['timestamp'].astype(int)\n\nprint(\"Wifi Data\")\ndisplay(sample_wifi.sample(3))\nprint(\"IBeacon Data\")\ndisplay(sample_beacon)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sample_acce = pd.DataFrame(sample_file.acce, columns = ['timestamp', 'acce_x', 'acce_y', 'acce_z'])\nsample_acce_uncali = pd.DataFrame(sample_file.acce_uncali, columns = ['timestamp', 'acce_x', 'acce_y', 'acce_z'])\nsample_gyro = pd.DataFrame(sample_file.gyro, columns = ['timestamp', 'gyro_x', 'gyro_y', 'gyro_z'])\nsample_gyro_uncali = pd.DataFrame(sample_file.gyro_uncali, columns = ['timestamp', 'gyro_x', 'gyro_y', 'gyro_z'])\nsample_magn = pd.DataFrame(sample_file.magn, columns = ['timestamp', 'magn_x', 'magn_y', 'magn_z'])\nsample_magn_uncali = pd.DataFrame(sample_file.magn_uncali, columns = ['timestamp', 'magn_x', 'magn_y', 'magn_z'])\nsample_ahrs = pd.DataFrame(sample_file.ahrs, columns = ['timestamp', 'ahrs_x', 'ahrs_y', 'ahrs_z'])\nsensor_df = sample_acce.copy()\n\nfor df in [sample_acce_uncali, sample_gyro, sample_gyro_uncali, sample_magn, sample_magn_uncali, sample_ahrs]:\n \n assert len(sensor_df) == len(sensor_df.merge(df, on = 'timestamp'))\n sensor_df = (sensor_df.merge(df, on = 'timestamp', suffixes = (\"\", \"_uncali\")))\n\nsensor_df['timestamp'] = sensor_df['timestamp'].astype(int)\n\nprint(\"Phone Sensors Data\")\ndisplay(sensor_df.sample(3))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"sample_waypoint = pd.DataFrame(sample_file.waypoint, columns = ['timestamp', 'x', 'y'])\nsample_waypoint['timestamp'] = sample_waypoint['timestamp'].astype(int)\n\nprint(\"Waypoint Data\")\ndisplay(sample_waypoint.sample(min(len(sample_waypoint), 3)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The 3 data sources (phone sensors, phone signals and waypoint) are not time aligned**. Check the following plot:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"all_timestamps = list(set(sample_waypoint.timestamp.tolist() + sensor_df.timestamp.tolist() +\n sample_wifi.ts_first_seen.tolist()+sample_wifi.ts_last_seen.tolist()+sample_beacon.timestamp.tolist()))\n\nall_timestamps_df = pd.DataFrame({'timestamp': all_timestamps}).sort_values('timestamp', ignore_index = True)\n\nall_timestamps_plus_data_df = (all_timestamps_df.merge(sample_waypoint[['timestamp']], how = 'left', indicator = True).rename({'_merge': 'waypoint'}, axis = 1)\n .replace({'left_only': 0, 'both': 1})\n .merge(sensor_df[['timestamp']], how = 'left', indicator = True).rename({'_merge': 'phone_sensors'}, axis = 1)\n .replace({'left_only': 2, 'both': 3})\n .merge(sample_wifi[['ts_first_seen']].rename({'ts_first_seen': 'timestamp'},axis=1), how = 'left', indicator = True)\n .rename({'_merge': 'wifi_first'}, axis = 1)\n .replace({'left_only': 4, 'both': 5})\n .merge(sample_wifi[['ts_last_seen']].rename({'ts_last_seen': 'timestamp'},axis=1), how = 'left', indicator = True)\n .rename({'_merge': 'wifi_last'}, axis = 1)\n .replace({'left_only': 6, 'both': 7})\n .merge(sample_beacon[['timestamp']], how = 'left', indicator = True).rename({'_merge': 'beacon'}, axis = 1)\n .replace({'left_only': 8, 'both': 9})\n .drop_duplicates(ignore_index = True))\n\nfig, ax = plt.subplots(1, 1, figsize = (16, 10))\nall_timestamps_plus_data_df.loc[all_timestamps_plus_data_df.timestamp > 1560914051576].head(2000).set_index('timestamp').plot(ax = ax)\nax.set_yticklabels(['on', 'off', 'on', 'off', 'on', 'off', 'on', 'off', 'on', 'off', 'on'])\nax.legend(loc='upper left', bbox_to_anchor=(1, 0.8))\nplt.suptitle('Each signal timestamp data: how the series are unaligned')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n### test and submission\n\nI have provided a visual explanation above of how test, train and submission data relate. For floors mapping check the [evaluation](https://www.kaggle.com/c/indoor-location-navigation/overview/evaluation) page. "},{"metadata":{"trusted":true},"cell_type":"code","source":"pretty(test_structure)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Differently from the training set we have just a list of paths, with no _site_ nor _floor_ information. \n\nLet's read one of the paths, `5694e13f4bb0bac39806b5ae`\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_test_path = read_data_file(os.path.join(test_path, '5694e13f4bb0bac39806b5ae.txt'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Everything is the same as for train data, except we don't have waypoint data (which is the target we are trying to predict). Let's get the sample submission data."},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/indoor-location-navigation/sample_submission.csv')\nsub[['site', 'path', 'timestamp']] = sub['site_path_timestamp'].str.split('_', expand=True)\ndisplay(sub.head(3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can already see we may be asked to predict floor, x and y coordinates for the same path at the same site for different timestamps. \n\nLet's now retrieve the corresponding path (`046cfa46be49fc10834815c6`) from test."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"@dataclass\nclass ReadDataDf:\n sensor_df: pd.DataFrame\n wifi: pd.DataFrame\n ibeacon: pd.DataFrame\n waypoint: pd.DataFrame\n\n\ndef read_data_file_df(data_filename):\n acce = []\n acce_uncali = []\n gyro = []\n gyro_uncali = []\n magn = []\n magn_uncali = []\n ahrs = []\n wifi = []\n ibeacon = []\n waypoint = []\n\n with open(data_filename, 'r', encoding='utf-8') as file:\n lines = file.readlines()\n\n for line_data in lines:\n line_data = line_data.strip()\n if not line_data or line_data[0] == '#':\n continue\n\n line_data = line_data.split('\\t')\n\n if line_data[1] == 'TYPE_WAYPOINT':\n waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n continue\n \n if line_data[1] == 'TYPE_ACCELEROMETER':\n acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n \n if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n \n if line_data[1] == 'TYPE_GYROSCOPE':\n gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n\n if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n \n if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n\n if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n\n if line_data[1] == 'TYPE_ROTATION_VECTOR':\n ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n continue\n\n if line_data[1] == 'TYPE_WIFI':\n sys_ts = line_data[0]\n ssid = line_data[2]\n bssid = line_data[3]\n rssi = line_data[4]\n lastseen_ts = line_data[6]\n wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n wifi.append(wifi_data)\n continue\n\n if line_data[1] == 'TYPE_BEACON':\n ts = line_data[0]\n uuid = line_data[2]\n major = line_data[3]\n minor = line_data[4]\n rssi = line_data[6]\n ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n ibeacon.append(ibeacon_data)\n continue\n \n def create_df(array, cols):\n try:\n return pd.DataFrame(array, columns = cols)\n except:\n return pd.DataFrame(columns = cols)\n \n acce = create_df(np.array(acce), cols = ['timestamp', 'acce_x', 'acce_y', 'acce_z'])\n acce_uncali = create_df(np.array(acce_uncali), cols = ['timestamp', 'acce_uncali_x', 'acce_uncali_y', 'acce_uncali_z'])\n gyro = create_df(np.array(gyro), cols = ['timestamp', 'gyro_x', 'gyro_y', 'gyro_z'])\n gyro_uncali = create_df(np.array(gyro_uncali), cols = ['timestamp', 'gyro_uncali_x', 'gyro_uncali_y', 'gyro_uncali_z'])\n magn = create_df(np.array(magn), cols = ['timestamp', 'magn_x', 'magn_y', 'magn_z'])\n magn_uncali = create_df(np.array(magn_uncali), cols = ['timestamp', 'magn_uncali_x', 'magn_uncali_y', 'magn_uncali_z'])\n ahrs = create_df(np.array(ahrs), cols = ['timestamp', 'ahrs_x', 'ahrs_y', 'ahrs_z'])\n \n sensor_df = acce.copy()\n for df in [acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs]:\n current_len = len(sensor_df)\n if len(df) == 0:\n continue\n sensor_df = (sensor_df.merge(df, on = 'timestamp', suffixes = (\"\", \"_uncali\")))\n assert current_len == len(sensor_df)\n \n sensor_df['timestamp'] = sensor_df['timestamp'].astype(int)\n \n wifi = create_df(wifi, cols = ['ts_last_seen', 'wifi_id_1', 'wifi_id_2', 'rssi', 'ts_first_seen'])\n wifi[['ts_first_seen', 'ts_last_seen']] = wifi[['ts_first_seen', 'ts_last_seen']].astype(int)\n ibeacon = create_df(ibeacon, cols = ['timestamp', 'beacon_id', 'rssi'])\n ibeacon['timestamp'] = ibeacon['timestamp'].astype(int)\n \n waypoint = create_df(np.array(waypoint), cols = ['timestamp', 'x', 'y'])\n waypoint['timestamp'] = waypoint['timestamp'].astype(int)\n \n return ReadDataDf(sensor_df, wifi, ibeacon, waypoint)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_path_046cfa46be49fc10834815c6 = read_data_file_df(os.path.join(test_path, '046cfa46be49fc10834815c6.txt'))\nsub_path_046cfa46be49fc10834815c6 = sub.loc[sub.path == '046cfa46be49fc10834815c6']\nsub_path_046cfa46be49fc10834815c6['timestamp'] = sub_path_046cfa46be49fc10834815c6['timestamp'].astype(int)\ndisplay(sub_path_046cfa46be49fc10834815c6)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"all_timestamps = list(set(test_path_046cfa46be49fc10834815c6.sensor_df.timestamp.tolist() + \n test_path_046cfa46be49fc10834815c6.ibeacon.timestamp.tolist() +\n #test_path_046cfa46be49fc10834815c6.wifi.ts_first_seen.tolist()+\n test_path_046cfa46be49fc10834815c6.wifi.ts_last_seen.tolist()))\n\nall_timestamps_df = pd.DataFrame({'timestamp': all_timestamps}).sort_values('timestamp', ignore_index = True)\n\nall_timestamps_plus_data_df = (all_timestamps_df\n .merge(test_path_046cfa46be49fc10834815c6.sensor_df[['timestamp']], how = 'left', indicator = True)\n .rename({'_merge': 'phone_sensors'}, axis = 1)\n .replace({'left_only': 2, 'both': 3})\n .merge(test_path_046cfa46be49fc10834815c6.wifi[['ts_last_seen']].rename({'ts_last_seen': 'timestamp'},axis=1), \n how = 'left', indicator = True)\n .rename({'_merge': 'wifi'}, axis = 1)\n .replace({'left_only': 4, 'both': 5})\n .merge(test_path_046cfa46be49fc10834815c6.ibeacon[['timestamp']], how = 'left', indicator = True).rename({'_merge': 'beacon'}, axis = 1)\n .replace({'left_only': 6, 'both': 7})\n .drop_duplicates(ignore_index = True))\n\nfig, ax = plt.subplots(1, 1, figsize = (16, 10))\nall_timestamps_plus_data_df.set_index('timestamp').plot(ax = ax)\nfor m, timestamp in enumerate(sub_path_046cfa46be49fc10834815c6.timestamp.tolist()):\n ax.axvline(timestamp, alpha = 0.5, ymin = 0, ymax = 10, linestyle = \":\", color = 'blue')\n ax.text(timestamp-1700, 7.3, \"prediction {}\".format(m+1), size = 10, alpha = 0.5, rotation = 30)\nax.set_yticklabels(['on', 'off', 'on', 'off', 'on', 'off', 'on', 'off', 'on', 'off', 'on'])\nax.legend(loc='upper left', bbox_to_anchor=(1, 0.8))\nplt.suptitle('Path vs Waypoint Timestamp for path 046cfa46be49fc10834815c6', fontsize = 20)\n#plt.text(x=100.8, y=7.4, s=\"When we are ask to predict waypoint, wrt to path data\", fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I would assume that for each path we can also use data from later timestamps to predict the waypoint. "},{"metadata":{"trusted":true},"cell_type":"markdown","source":"My ideas end here, I'll update the notebook in the next few weeks if anything comes to my mind. Please tell me what you think!\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}