{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Indoor location and navigation competition EDA"},{"metadata":{},"cell_type":"markdown","source":"Starting EDA to understand the data available and building up from other great works that I have seen on the public space (please upvote [this one ](https://www.kaggle.com/andradaolteanu/indoor-navigation-complete-data-understanding/data#notebook-container), [this one](https://www.kaggle.com/iamleonie/intro-to-indoor-location-navigation) and [this one](https://www.kaggle.com/c/indoor-location-navigation/discussion/215445))"},{"metadata":{},"cell_type":"markdown","source":"## The task  \nGiven a path file, the goal is to predict the floor and the waypoint location (x,y) at the timestamp provided in the `sample_submission.csv` file."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport os\nimport json\n\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nimport matplotlib.dates as mdates\n\nfrom datetime import datetime\nfrom functools import reduce\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Understanding the input data  \n- __train__ : path to training data, organized by site and floor. Each file within each floor folder contains the data relative to a single path on that floor.\n- __test__ : path to test examples. Each file here represents a single path on a single floor, and it does not have the waypoint (x,y) data since that is part of what we want to predict\n- __metadata__ : organized by site and floor, and containing for each of them `floor_image.png`, `floor_info.json` and `geojson_map.json`\n\n\nFor each example, the following signals are recorded at each position:\n- accelerometer\n- magnetic field\n- gyroscope\n- rotation vector\n- WiFi\n- Bluetooth iBeacon\n- waypoint locations (x,y)\n- each example belongs to a particolar floor of a particular building"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = \"../input/indoor-location-navigation/train\"\ntest_path = \"../input/indoor-location-navigation/test\"\nsites_path = \"../input/indoor-location-navigation/metadata\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#checking the number of files in training\n_, train_dir_names, _ = next(os.walk(train_path))\n\ntrain_filenames = []\nn_floors = 0\nfor tdn in train_dir_names: #for each training example folder (so for each site)\n    _, sub_dir, _ = next(os.walk(train_path + '/' + tdn))\n    n_floors += len(sub_dir)\n    for sd in sub_dir: #for each sub-folder (so for each floor of the site)\n        _, _, filenames = next(os.walk(train_path + '/' + tdn + '/' + sd)) #list all the files\n    \n        train_filenames += filenames\n        \nprint(\"There are {} training examples over a total of {} sites and {} floors\".format(len(train_filenames),\n                                                                         len(train_dir_names),\n                                                                                    n_floors))\n\n#checking the number of test examples\n_, test_dir_names, tst_filenames = next(os.walk(test_path))\n\nprint(\"There are {} test examples\".format(len(tst_filenames)))\n\n#counting the sites for which metadata is provided\n_, meta_dir_names, _ = next(os.walk(sites_path))\nprint(\"Metadata provided for {} sites\".format(len(meta_dir_names)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Using the code from the competition repo \n\nRe-structuring the data from scratch can be a serious pain since the single example above has 14210 lines and these lines vary from example to example.  \nWe can instead download the [competition repo](https://github.com/location-competition/indoor-location-competition-20) and then upload it as dataset Using `cp -r path/* ./` . In this way we can accees the code of the repo and use `read_data_file` in the `io_f` to read information correctly"},{"metadata":{"trusted":true},"cell_type":"code","source":"!cp -r ../input/indoorlocationcompetition20master/indoor-location-competition-20-master/* ./","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from io_f import read_data_file\n\n#input example\nexample_path = \"../input/indoor-location-navigation/train/5cd56b83e2acfd2d33b5cab0/2F/5cf61eed731d5200089a627f.txt\"\n\n\nexample = read_data_file(example_path)\n\n#list of the fields as per read_data_file class in io_f\nprint(\"'acce' refers to the accelerometer data - shape {}\".format(example.acce.shape))\nprint(\"'acce_uncali' refers to the uncalibrated accelerometer data - shape {}\".format(example.acce_uncali.shape))\nprint(\"'gyro' refers to the gyroscope data - shape {}\".format(example.gyro.shape))\nprint(\"'gyro_uncali' refers to the uncalibrated gyroscope data - shape {}\".format(example.gyro.shape))\nprint(\"'magn' refers to the magnetic field data - shape {}\".format(example.magn.shape))\nprint(\"'magn_uncali' refers to the uncalibrated magnetic field data - shape {}\".format(example.magn_uncali.shape))\nprint(\"'ahrs' refers to the rotation vector data - shape {}\".format(example.ahrs.shape))\nprint(\"'wifi' refers to the wifi data - shape {}\".format(example.wifi.shape))\nprint(\"'ibeacon' refers to the bluetooth data - shape {}\".format(example.ibeacon.shape))\nprint(\"'waypoint' refers to the x,y position coordinates - shape {}\".format(example.waypoint.shape))\n#    waypoint = np.array(waypoint)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"[Here](https://www.kaggle.com/iamleonie/intro-to-indoor-location-navigation#Inertial-Measurement-Unit-(IMU)) is a great explanation of Inertial Measurement Unit (IMU) data, consisting of signals coming from **accelerometer**, **Gyroscopes** and **Magnetometers**. For them, as you can see above, the data shape is the same.  \nThe first item is usually a timestamp in milliseconds (so it will be converted to datetime using pandas.fromtimestamp inputting the value divided by 1000."},{"metadata":{"trusted":true},"cell_type":"code","source":"#example of waypoint data\nexmp_waypoint_df = pd.DataFrame(example.waypoint, columns = ['timestamp', 'x', 'y'])\nexmp_waypoint_df['timestamp'] = exmp_waypoint_df['timestamp'].apply(lambda x: datetime.fromtimestamp(x/1000))\nexmp_waypoint_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#example of accelerometer data (gyroscopes, magnetometers (and their uncalibrated version)and rotation vector can be managed in the same way)\nexmp_acce_df = pd.DataFrame(example.acce, columns = ['timestamp', 'x', 'y', 'z'])\nexmp_acce_df['timestamp'] = exmp_acce_df['timestamp'].apply(lambda x: datetime.fromtimestamp(x/1000))\nexmp_acce_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#merging IMU data together\ndef IMU_df_gen(obs_path, datetimeconvert = True):\n    \n    \"\"\"\n    This function creates a dataframe with IMU data for a single observation.\n    \"\"\"\n    \n    obs = read_data_file(obs_path)\n    \n    acce = pd.DataFrame(obs.acce, columns = ['timestamp', 'acce_x', 'acce_y', 'acce_z'])\n    acce_uncali = pd.DataFrame(obs.acce_uncali, columns = ['timestamp', 'acce_uncali_x', 'acce_uncali_y', 'acce_uncali_z'])\n    \n    gyro = pd.DataFrame(obs.gyro, columns = ['timestamp', 'gyro_x', 'gyro_y', 'gyro_z'])\n    gyro_uncali = pd.DataFrame(obs.gyro_uncali, columns = ['timestamp', 'gyro_uncali_x', 'gyro_uncali_y', 'gyro_uncali_z'])\n    \n    magn = pd.DataFrame(obs.magn, columns = ['timestamp', 'magn_x', 'magn_y', 'magn_z'])\n    magn_uncali = pd.DataFrame(obs.magn_uncali, columns = ['timestamp', 'magn_uncali_x', 'magn_uncali_y', 'magn_uncali_z'])\n    \n    ahrs = pd.DataFrame(obs.ahrs, columns = ['timestamp', 'ahrs_x', 'ahrs_y', 'ahrs_z'])\n    \n    #merging the dfs\n    dfs = [acce, acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs]\n    \n    IMU_df = reduce(lambda  left,right: pd.merge(left,right,on=['timestamp'], how='outer'), dfs)\n    \n    \n    if datetimeconvert:\n        IMU_df['timestamp'] = IMU_df.timestamp.apply(lambda x: datetime.fromtimestamp(int(x)/1000))\n    \n    return IMU_df\n\nIMU_df = IMU_df_gen(example_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, acce_axs = plt.subplots(nrows=3, ncols=1)\n\nplt.subplots_adjust(left=None, bottom=1, right=None, top=2, wspace=None, hspace=None)\n\nacce_x_plot = IMU_df.plot(x = 'timestamp', y=['acce_x', 'acce_uncali_x'], figsize = (20,3), title = 'acce_x vs acce_uncali_x', ax = acce_axs[0])\nacce_y_plot = IMU_df.plot(x = 'timestamp', y=['acce_y', 'acce_uncali_y'], figsize = (20,3), title = 'acce_y vs acce_uncali_y', ax = acce_axs[1])\nacce_y_plot = IMU_df.plot(x = 'timestamp', y=['acce_z', 'acce_uncali_z'], figsize = (20,3), title = 'acce_z vs acce_uncali_z', ax = acce_axs[2])\n        \n\n\nfor a in range(len(acce_axs)):\n    if a < 2:\n        acce_axs[a].axes.get_xaxis().set_visible(False)\n    else:\n        acce_axs[a].xaxis.set_major_locator(mdates.SecondLocator(interval = 5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, magn_axs = plt.subplots(nrows=3, ncols=1)\n\nplt.subplots_adjust(left=None, bottom=1, right=None, top=2, wspace=None, hspace=None)\n\nmagn_x_plot = IMU_df.plot(x = 'timestamp', y=['magn_x', 'gyro_uncali_x'], figsize = (20,3), title = 'magn_x vs magn_uncali_x', ax = magn_axs[0])\nmagn_y_plot = IMU_df.plot(x = 'timestamp', y=['magn_y', 'gyro_uncali_y'], figsize = (20,3), title = 'magn_y vs magn_uncali_y', ax = magn_axs[1])\nmagn_y_plot = IMU_df.plot(x = 'timestamp', y=['magn_z', 'gyro_uncali_z'], figsize = (20,3), title = 'magn_z vs magn_uncali_z', ax = magn_axs[2])\nfor a in range(len(magn_axs)):\n    if a < 2:\n        magn_axs[a].axes.get_xaxis().set_visible(False)\n    else:\n        magn_axs[a].xaxis.set_major_locator(mdates.SecondLocator(interval = 5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, gyro_axs = plt.subplots(nrows=3, ncols=1)\n\nplt.subplots_adjust(left=None, bottom=1, right=None, top=2, wspace=None, hspace=None)\n\ngyro_x_plot = IMU_df.plot(x = 'timestamp', y=['gyro_x', 'gyro_uncali_x'], figsize = (20,3), title = 'gyro_x vs gyro_uncali_x', ax = gyro_axs[0])\ngyro_y_plot = IMU_df.plot(x = 'timestamp', y=['gyro_y', 'gyro_uncali_y'], figsize = (20,3), title = 'gyro_y vs gyro_uncali_y', ax = gyro_axs[1])\ngyro_y_plot = IMU_df.plot(x = 'timestamp', y=['gyro_z', 'gyro_uncali_z'], figsize = (20,3), title = 'gyro_z vs gyro_uncali_z', ax = gyro_axs[2])\nfor a in range(len(gyro_axs)):\n    if a < 2:\n        gyro_axs[a].axes.get_xaxis().set_visible(False)\n    else:\n        gyro_axs[a].xaxis.set_major_locator(mdates.SecondLocator(interval = 5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FLOORS MAPPING  \nI am considering the '**LG' floors as 'Lower ground**' so they will be the same as the '**B**' floors.  \nSame for '**BF**' (basement floor).  \nP levels are also usually referred to parkings below the ground floor.  \nI have interpreted **'G' as ground floor** so this will be same as **'F1'**. \n\n*Classifying BM (I guess basement mezzanine), M (mezzanine) and LM is probably more complicated\nI'll just probably drop them (giving value 99 in the dictionary for now)*"},{"metadata":{"trusted":true},"cell_type":"code","source":"#grouping the floors by type and counting them\n\nfloortypes = []\n\nfor tdn in train_dir_names: #for each training example folder (so for each site)\n    _, sub_dir, _ = next(os.walk(train_path + '/' + tdn))\n    for sd in sub_dir: #for each sub-folder (so for each floor of the site)\n        floortypes.append(sd)\n\nfloors = pd.Series(floortypes)\n        \nplt.figure(figsize = (20,10))\nsns.countplot(floors, color = 'lightblue')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"floors.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#floors mapping\n\nfloors_dict = {'B3': -3,\n               'B2': -2, 'LG2':-2, 'P2':-2,\n               'B1': -1, 'LG1':-1, 'P1':-1, 'B':-1, 'BF':-1,\n              'F1':0, '1F':0, 'L1':0, 'G':0,\n              'F2':1, '2F':1, 'L2':1,\n              'F3':2, '3F':2, 'L3':2,\n              'F4':3, '4F':3, 'L4':3,\n              'F5':4, '5F':4, 'L5':4,\n              'F6':5, '6F':5, 'L6':5,\n              'F7':6, '7F':6, 'L7':6,\n              'F8':7, '8F':7, 'L8':7,\n              'F9':8, '9F':8, 'L9':8,\n              'F10':9, '10F':9, 'L10':9,\n              'L11':10,\n              'LM':99, 'M':99, 'BM':99}\n\nfloors_mapped = floors.apply(lambda x:floors_dict[x])\n\nplt.figure(figsize = (20,10))\nsns.countplot(floors_mapped, color='lightblue')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing a single path from a single example"},{"metadata":{"trusted":true},"cell_type":"code","source":"from visualize_f import visualize_trajectory, visualize_heatmap\n\nexmp_trajectory = example.waypoint #this returns timestamp, x,y\nexmp_trajectory = exmp_trajectory[:, 1:] #keeping only coords x,y\nexmp_base_path = \"/\".join(example_path.split(\"/\")[:4]).replace(\"/train\", \"/metadata\")\nexmp_site = example_path.split(\"/\")[4] #getting the site folder\nexmp_floor = example_path.split(\"/\")[5] #getting the floor\n\nexmp_floor_img = \"{}/{}/{}/floor_image.png\".format(exmp_base_path, exmp_site, exmp_floor)\n\nexmp_floor_json = \"{}/{}/{}/floor_info.json\".format(exmp_base_path, exmp_site, exmp_floor)\nwith open(exmp_floor_json) as jsonf:\n    json_data = json.load(jsonf)\n    \nwidth_m = json_data['map_info']['width']\nheight_m = json_data['map_info']['height']\n\nvisualize_trajectory(trajectory = exmp_trajectory,\n                     floor_plan_filename = exmp_floor_img,\n                     width_meter = width_m,\n                     height_meter = height_m,\n                     title = 'single path from single example')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Visualizing the magnetic strength on a single path"},{"metadata":{"trusted":true},"cell_type":"code","source":"from main import calibrate_magnetic_wifi_ibeacon_to_position\nfrom main import extract_magnetic_strength\n\n# Extracting the magnetic strength\nmwifi_data = calibrate_magnetic_wifi_ibeacon_to_position([example_path])\nmagnetic_strength = extract_magnetic_strength(mwifi_data)\n\nheat_positions = np.array(list(magnetic_strength.keys()))\nheat_values = np.array(list(magnetic_strength.values()))\n\nvisualize_heatmap(heat_positions,\n                  heat_values,\n                     floor_plan_filename = exmp_floor_img,\n                     width_meter = width_m,\n                     height_meter = height_m,\n                     title = 'magnetic strength from single example')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing WiFi access points from a single example"},{"metadata":{"trusted":true},"cell_type":"code","source":"from main import extract_wifi_rssi, extract_wifi_count\n\n# Get WiFi data\nwifi_rssi = extract_wifi_rssi(mwifi_data)\nprint(f'This floor has {len(wifi_rssi.keys())} wifi aps (access points).')\n\nwifi_counts = extract_wifi_count(mwifi_data)\nheat_positions = np.array(list(wifi_counts.keys()))\nheat_values = np.array(list(wifi_counts.values()))\n# filter out positions that no wifi detected\nmask = heat_values != 0\nheat_positions = heat_positions[mask]\nheat_values = heat_values[mask]\n\n# The heatmap\nvisualize_heatmap(heat_positions, \n                  heat_values, \n                  floor_plan_filename = exmp_floor_img,\n                    width_meter = width_m,\n                    height_meter = height_m,\n                  colorbar_title='count', \n                  title='wifi access on a single example')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}