{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook, I will show you the approach I have used to generate the WIFI train and test datasets:\n* The training dataset will contain (x,y) position, floor and wifi features (bssid + rssi) for every wifi timestamp.\n* The testing dataset will contain the wifi features for every wifi timestamp.\n\n\nHowever everyone has noticed that the ground truth locations and WIFI data (also accelerometer, gyro, ...) are provided at different timestamps.\nConsequently, we are not able to associate straightaway the ground truth with any sensor data. We cannot predict with 100% accuracy the position where the sensor data were recorded so we need to make some assumptions.\n\n\nFirst, I have made few observations:\n* The smartphone is held by a person so the walking speed is limited.\n* Two consecutive waypoints are only few meters apart. Also, given the limited time between these two waypoints (generally few seconds), the probability of a detour is very low.\n* Any sensor data recorded at a time between the two waypoint timestamps should be close enough to these two points.\n\nBased on that, I have assumed the person holding the smartphone is walking straight between any two consecutive waypoints and at a constant pace. Hence, knowing two waypoint locations and their associated timestamps, we can derive the sensor locations based on the sensor timestamps using a simple **linear interpolation.**\n\nUsing this approach, we are able to have an estimated (x,y) position for every WIFI timestamp.\n\nI was able to get a score around 11 on the Leaderboard with WIFI features + Random Forest model. I have not made any hyperparameters fine-tuning or used a more advanced model at this stage; so there is lot of room for improvement.\n\nHere below, I will present the code I have made to generate those datasets. I will also provide in a separate notebook the files created as the code takes several hours to run.\n","metadata":{}},{"cell_type":"markdown","source":"# Import libraries","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\nfrom dataclasses import dataclass\n\nimport matplotlib.pyplot as plt # visualization\nplt.rcParams.update({'font.size': 14})\nimport seaborn as sns # visualization\n\nimport warnings # Supress warnings \nwarnings.filterwarnings('ignore')\n\nfrom tqdm import tqdm\n\nimport json\nimport plotly.graph_objs as go\nfrom PIL import Image","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Function to read data","metadata":{}},{"cell_type":"code","source":"# copy from https://github.com/location-competition/indoor-location-competition-20/blob/master/io_f.py\n\n@dataclass\nclass ReadData:\n    acce: np.ndarray\n    acce_uncali: np.ndarray\n    gyro: np.ndarray\n    gyro_uncali: np.ndarray\n    magn: np.ndarray\n    magn_uncali: np.ndarray\n    ahrs: np.ndarray\n    wifi: np.ndarray\n    ibeacon: np.ndarray\n    waypoint: np.ndarray\n\n\ndef read_data_file(data_filename):\n    acce = []\n    acce_uncali = []\n    gyro = []\n    gyro_uncali = []\n    magn = []\n    magn_uncali = []\n    ahrs = []\n    wifi = []\n    ibeacon = []\n    waypoint = []\n\n    with open(data_filename, 'r', encoding='utf-8') as file:\n        lines = file.readlines()\n\n    for line_data in lines:\n        line_data = line_data.strip()\n        if not line_data or line_data[0] == '#':\n            continue\n\n        line_data = line_data.split('\\t')\n\n        if line_data[1] == 'TYPE_WAYPOINT':\n            waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n            continue\n       \n        if line_data[1] == 'TYPE_ACCELEROMETER':\n            acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n            acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_GYROSCOPE':\n            gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n            gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n        \n        if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n            magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n            magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ROTATION_VECTOR':\n            ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_WIFI':\n            sys_ts = line_data[0]\n            ssid = line_data[2]\n            bssid = line_data[3]\n            rssi = line_data[4]\n            lastseen_ts = line_data[6]\n            wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n            wifi.append(wifi_data)\n            continue\n\n        if line_data[1] == 'TYPE_BEACON':\n            ts = line_data[0]\n            uuid = line_data[2]\n            major = line_data[3]\n            minor = line_data[4]\n            rssi = line_data[6]\n            ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n            ibeacon.append(ibeacon_data)\n            continue\n        \n    \n    acce = np.array(acce)\n    acce_uncali = np.array(acce_uncali)\n    gyro = np.array(gyro)\n    gyro_uncali = np.array(gyro_uncali)\n    magn = np.array(magn)\n    magn_uncali = np.array(magn_uncali)\n    ahrs = np.array(ahrs)\n    wifi = np.array(wifi)\n    ibeacon = np.array(ibeacon)\n    waypoint = np.array(waypoint)\n    \n    return ReadData(acce, acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs, wifi, ibeacon, waypoint)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate wifi train dataset","metadata":{}},{"cell_type":"code","source":"# read sample submission file\nssubm = pd.read_csv(f\"../input/indoor-location-navigation/sample_submission.csv\")\nssubm['site'] = ssubm['site_path_timestamp'].apply(lambda x: x.split('_')[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The code below generates the training WIFI datasets for all 24 buildings in the sample_submission.csv file.","metadata":{}},{"cell_type":"code","source":"#-----------------------------------------------------\n#             STEP 1: generate df_wp_wifi\n#-----------------------------------------------------\n\n\"\"\"\nGenerate a dataframe for each site that contains estimated wifi waypoint and wifi data for every wifi timestamp.\n\"\"\"\n\nfor siteID in ssubm.site.unique():\n  print(siteID)\n\n  df_wp_wifi = pd.DataFrame()\n  for floor in os.listdir(f\"../input/indoor-location-navigation/train/{siteID}\"):\n\n      for path in os.listdir(f\"../input/indoor-location-navigation/train/{siteID}/{floor}/\"):\n\n          sample_file = read_data_file(f\"../input/indoor-location-navigation/train/{siteID}/{floor}/{path}\")\n\n          waypoint_df = pd.DataFrame(sample_file.waypoint)\n          waypoint_df.columns = ['sys_ts', 'x', 'y']\n          waypoint_df.sys_ts = waypoint_df.sys_ts.astype('int')\n          waypoint_df['type'] = 'WAYPOINT'\n\n\n          if sample_file.wifi.shape[0] != 0:\n              wifi_df = pd.DataFrame(sample_file.wifi)\n              wifi_df.columns = ['sys_ts', 'ssid', 'bssid', 'rssi', 'lastseen_ts']\n              wifi_df.sys_ts = wifi_df.sys_ts.astype('int')\n              wifi_df.rssi = wifi_df.rssi.astype('int')\n              wifi_df['type'] = 'WIFI'\n\n              concat = pd.concat([waypoint_df, wifi_df])\n              concat = concat.sort_values(by=['sys_ts'])\n              concat = concat.set_index('sys_ts')\n              concat = concat.interpolate(method='index', axis=0).reset_index()\n            \n              concat['floor'] = floor\n              concat['path'] = path\n              concat = concat[['sys_ts', 'path', 'floor', 'type', 'x', 'y',  'ssid', 'bssid', 'rssi', 'lastseen_ts']]\n\n          else:\n              concat = waypoint_df\n              concat = concat.sort_values(by=['sys_ts'])\n              concat = concat.set_index('sys_ts')\n              # fill (x,y) nan values using interpolate method on index (sys_ts)\n              concat = concat.interpolate(method='index', axis=0).reset_index()\n              concat['floor'] = floor\n              concat['path'] = path\n\n\n\n          df_wp_wifi = pd.concat([df_wp_wifi, concat])\n\n  # floor encoding\n  df_wp_wifi['floor_enc'] = df_wp_wifi['floor'].replace(\n      {\"B3\":-3,\"B2\":-2,\"B1\":-1,\"F1\":0,\"1F\":0,\"F2\":1,\"2F\":1,\"F3\":2,\"3F\":2,\"F4\":3,\"4F\":3,\n            \"F5\":4,\"5F\":4,\"F6\":5,\"6F\":5,\"F7\":6,\"7F\":6,\"F8\":7,\"8F\": 7,\"F9\":8,\"9F\":8,\"F10\":9}\n  )\n\n  # bssid occurence >= 1000 saved in a list\n  top_bssid = df_wp_wifi['bssid'].value_counts()[df_wp_wifi['bssid'].value_counts() >= 1000].index\n\n  # filter df to keep only rows with bssid > 1000 occurence\n  df_wp_wifi = df_wp_wifi[df_wp_wifi.bssid.isin(top_bssid)]\n\n  # Drop duplicates\n  df_wp_wifi_dup = df_wp_wifi.drop_duplicates(subset=['sys_ts', 'path', 'bssid'])\n\n  del df_wp_wifi\n\n  # drop columns\n  df_wp_wifi_dup = df_wp_wifi_dup.drop(columns=['floor', 'type', 'ssid', 'lastseen_ts'])\n\n  df_wp_wifi_dup.to_csv(f\"./train_datasets/df_wp_wifi_dup_{siteID}_bssid_1000.csv\")\n\n\n  #-----------------------------------------------------\n  #             STEP 2: generate wifi features using bssid and rssi columns for every timestamp\n  #-----------------------------------------------------\n\n\n  df4 = pd.DataFrame()\n  df5 = pd.DataFrame()\n\n  for i, path in enumerate(df_wp_wifi_dup['path'].unique()):\n\n      for ts in df_wp_wifi_dup[(df_wp_wifi_dup.path == path)].sys_ts.unique():\n\n          bssid = df_wp_wifi_dup[(df_wp_wifi_dup.path == path) & (df_wp_wifi_dup.sys_ts==ts)].reset_index(drop=True)\n            \n          \n          col_bssid = bssid.bssid.tolist()\n          bssid_dummies = pd.get_dummies(bssid.bssid)[col_bssid]\n\n          s = bssid.rssi  # store rssi values in a serie\n          df = pd.DataFrame(0, index=s.index, columns=s.index, dtype=s.dtype)  # df with O values\n          np.fill_diagonal(df.values, s) #fill diagonal with rssi values\n\n          # multiply bssid dummy matrix with rssi diagonal matrix. This will just replace ones from the bssid_dummies matrix with corresponding rssi values. \n          # In other words, I have combined bssid and rssi features into one single feature.\n          df_temp = pd.DataFrame(np.array(bssid_dummies).dot(np.array(df)), columns=col_bssid).reset_index(drop=True)\n        \n          df1 = pd.concat([bssid, df_temp], axis=1)[['sys_ts', 'x', 'y', 'floor_enc']].groupby('sys_ts').mean().reset_index()\n          df2 = pd.concat([bssid, df_temp], axis=1).drop(columns=['path', 'x', 'y', 'bssid', 'rssi', 'floor_enc']).groupby('sys_ts').sum().reset_index()\n          \n          df3 = df1.merge(df2, how='left').reset_index(drop=True)\n          df3['path'] = path\n\n          df4 = pd.concat([df4, df3], axis=0, ignore_index=True).reset_index(drop=True)\n\n\n      df5 = pd.concat([df5, df4], axis=0)\n\n  # drop duplicates\n  df5_final = df5.drop_duplicates(subset=['path', 'sys_ts'])\n    \n  del df5\n    \n#-----------------------------------------------------\n#             STEP 3: save final dataset\n#-----------------------------------------------------\n\n\n  # save df in a csv file\n  df5_final.to_csv(f\"../input/indoor-location-navigation/train_datasets/{siteID}_bssid_1000.csv\")\n\n  del df5_final","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate wifi test dataset","metadata":{}},{"cell_type":"code","source":"\n#-----------------------------------------------------\n#             STEP 1: generate df_wifi\n#-----------------------------------------------------\n\nfor siteID in test_sites:\n  print(siteID)\n\n  df_wifi = pd.DataFrame()\n  for path in os.listdir(f\"../input/indoor-location-navigation/test/{siteID}\"):\n\n    sample_file = read_data_file(f\"../input/indoor-location-navigation/test/{siteID}/{path}\")\n\n    if sample_file.wifi.shape[0] != 0:\n        wifi_df = pd.DataFrame(sample_file.wifi)\n        wifi_df.columns = ['sys_ts', 'ssid', 'bssid', 'rssi', 'lastseen_ts']\n        wifi_df.sys_ts = wifi_df.sys_ts.astype('int')\n        wifi_df.rssi = wifi_df.rssi.astype('int')\n\n        wifi_df = wifi_df.sort_values(by=['sys_ts'])\n        wifi_df['path'] = path\n        wifi_df = wifi_df[['sys_ts', 'path', 'ssid', 'bssid', 'rssi', 'lastseen_ts']]\n\n    else:\n      wifi_df = pd.DataFrame(columns=['sys_ts', 'path', 'ssid', 'bssid', 'rssi', 'lastseen_ts'])\n\n    df_wifi = pd.concat([df_wifi, wifi_df])\n\n  col_train_bssid = pd.read_csv(f\"../input/indoor-location-navigation/train_datasets/{siteID}_bssid_1000.csv\").drop(columns=['Unnamed: 0', 'sys_ts', 'path', 'x', 'y', 'floor_enc']).columns\n\n  # filter df to keep only rows with bssid > 1000 occurence\n  df_wifi = df_wifi[df_wifi.bssid.isin(col_train_bssid)]\n\n  # Drop duplicates\n  df_wifi_dup = df_wifi.drop_duplicates(subset=['sys_ts', 'path', 'bssid'])\n  del df_wifi\n\n  # drop columns\n  df_wifi_dup = df_wifi_dup.drop(columns=['ssid', 'lastseen_ts'])\n\n  df_wifi_dup.to_csv(f\"../input/indoor-location-navigation/test_datasets/df_wifi_dup_test_{siteID}_bssid_1000.csv\")\n\n\n  #-----------------------------------------------------\n  #             STEP 2: generate wifi features using bssid and rssi columns for every timestamp\n  #-----------------------------------------------------\n\n\n  df2 = pd.DataFrame()\n  df3 = pd.DataFrame()\n\n  for i, path in enumerate(df_wifi_dup['path'].unique()):\n      print(f\"{i}/{len(df_wifi_dup['path'].unique())}\")\n      for ts in df_wifi_dup[(df_wifi_dup.path == path)].sys_ts.unique():\n\n          bssid = df_wifi_dup[(df_wifi_dup.path == path) & (df_wifi_dup.sys_ts==ts)].reset_index(drop=True)\n\n          col_bssid = bssid.bssid.tolist()\n          bssid_dummies = pd.get_dummies(bssid.bssid)[col_bssid]\n\n          s = bssid.rssi\n          df = pd.DataFrame(0, index=s.index, columns=s.index, dtype=s.dtype)\n          np.fill_diagonal(df.values, s) #fill diagonal with rssi values\n\n          df_temp = pd.DataFrame(np.array(bssid_dummies).dot(np.array(df)), columns=col_bssid).reset_index(drop=True)\n\n          # make sure training and test csv files have the same wifi columns\n          for col in col_train_bssid:\n            if col not in col_bssid:\n              df_temp[col] = np.nan\n            \n          df1 = pd.concat([bssid, df_temp], axis=1).drop(columns=['path', 'bssid', 'rssi']).groupby('sys_ts').sum().reset_index()\n\n          # df3 = df1.merge(df2, how='left').reset_index(drop=True)\n          df1['path'] = path\n\n          df2 = pd.concat([df2, df1], axis=0, ignore_index=True).reset_index(drop=True)\n\n\n      df3 = pd.concat([df3, df2], axis=0)\n\n  # drop duplicates\n  df3_final = df3.drop_duplicates(subset=['path', 'sys_ts'])\n    \n  del df3\n\n#-----------------------------------------------------\n#             STEP 3: save final dataset\n#-----------------------------------------------------\n\n  # # save df in a csv file\n  df3_final.to_csv(f\"../input/indoor-location-navigation/test_datasets/test_{siteID}_bssid_1000.csv\")\n\n  del df3_final","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}