![](\"https://storage.googleapis.com/kaggle-competitions/kaggle/22559/logos/header.png?t=2020-09-30-17-40-59\")
"},{"metadata":{},"cell_type":"markdown","source":"# 1. Welcome to The competition\n\n**By Microsoft Research***\n- Accurate indoor positioning, based on public sensors and user permission, allows for a great location-based experience even when we aren’t outside.\n- Current positioning solutions have poor accuracy, particularly in multi-level buildings, or generalize poorly to small datasets. Additionally, GPS was built for a time before smartphones. Today’s use cases often require more granularity than is typically available indoor.\n- Task is to predict the ***indoor position of smartphones based on real-time sensor data***, provided by indoor positioning technology company **XYZ10 in partnership with Microsoft Research**. we'll locate devices using ***“active” localization data***, which is made available with the cooperation of the user. Unlike passive localization methods (e.g. radar, camera), the data provided for this competition requires explicit user permission. \n- Dataset comprised of nearly 30,000 traces from over 200 buildings."},{"metadata":{},"cell_type":"markdown","source":"# 1.1 Evaluation Criteria and Metrics"},{"metadata":{},"cell_type":"markdown","source":"Submissions are evaluated on the `mean position error` as defined as:\n\n$$\\text{mean position error} = \\frac{1}{N} \\sum_{i=1}^{N} \n \\left( \\sqrt{( \\hat{x}_i - x_i )^{2} + ( \\hat{y}_i - y_i )^{2}} \n + p \\cdot | \\hat{f}_{i} - f_i | \\right)$$\nwhere:\n\n- $N$ is the number of rows in the test set \n- $\\hat{x}_i, \\hat{y}_i$ are the predicted locations for a given test row\n- $x_i, y_i$ are the ground truth locations for a given test row\n- $p$ is the floor penalty, set at 15\n- $\\hat{f}_{i}, f_{i}$ are the predicted and ground truth integer floor level for a given test row\n\nIMPORTANT: The integer `floor` used in the submission must be mapped from the char/int floors used in the dataset. The mapping is as follows:\n\n- F1, 1F → 0\n- F2, 2F → 1\netc.\n- B1, 1B → -1\n- B2, 2B → -2\n\nThere are other floor names in the training data, e.g., LG2, LM, etc., which you may decide to use for training, but none of these non-standard floors are found in the test set."},{"metadata":{},"cell_type":"markdown","source":"# 1.2 Submission Format"},{"metadata":{},"cell_type":"markdown","source":"For each `site_path_timestamp` row in the test set, we must predict the floor converted to an integer as per above and the `x` and `y` of the waypoint. The file should contain a header and have the following format:\n\n```\nsite_path_timestamp,floor,x,y\n5a0546857ecc773753327266_046cfa46be49fc10834815c6_1578474564146,0,15.0,55.0\n5a0546857ecc773753327266_046cfa46be49fc10834815c6_1578474573154,0,25.0,65.0\n5a0546857ecc773753327266_046cfa46be49fc10834815c6_1578474579463,0,35.0,75.0\netc.\n```"},{"metadata":{},"cell_type":"markdown","source":"# 2. Data"},{"metadata":{},"cell_type":"markdown","source":"- The dataset for this competition consists of dense indoor signatures of WiFi, geomagnetic field, iBeacons etc., as well as ground truth (waypoint) (locations) collected from hundreds of buildings in Chinese cities.\n- The data found in path trace files (*.txt) corresponds to an indoor path between position p_1 and p_2 walked by a site-surveyor.\n\n# 2.1 The Data Collection Process\n\n- During the walk, an Android smartphone is held flat in front of the surveyors body, and a sensor data recording app is running on the device to collect IMU (accelerometer, gyroscope) and geomagnetic field (magnetometer) readings, as well as WiFi and Bluetooth iBeacon scanning results.\n- A detailed description of the format of trace file is shown, along with other details and processing scripts, at this [github link](https://github.com/location-competition/indoor-location-competition-20).\n- In addition to raw trace files, floor plan metadata (e.g., raster image, size, GeoJSON) are also included for each floor.\n\nIn the training files, you may find occasionally that a line is missing the ending newline character, causing it to run on to the next line. It is up to you how you want to handle this issue. This issue is not found in the test data.
\n\n# 2.2 Files\n\n- **train** - training path files, organized by site and floor; each path files contains the data of a single path on a single floor\n- **test** - test path files, organized by site and floor; each path files contains the data of a single path on a single floor, but without the waypoint (x, y) data; the task of this competition is, for a given site-path file, predict the floor and waypoint locations at the timestamps given in the sample_submission.csv file\n- **metadata** - floor metadata folder, organized by site and floor, which includes the following for each floor:\n - floor_image.png\n - floor_info.json\n - geojson_map.json\n- **sample_submission.csv** - a sample submission file in the correct format; each has a unique id which contains a site id, a path id, and the timestamp within the trace for which to make a prediction; see the Evaluation page for the required integer mapping of floor names"},{"metadata":{},"cell_type":"markdown","source":"# 3 EDA"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport json\nimport glob\nimport random\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = \"../input/indoor-location-navigation/train\"\ntest_dir = \"../input/indoor-location-navigation/test\"\nmeta_dir = \"../input/indoor-location-navigation/metadata\"\nss = \"../input/indoor-location-navigation/sample_submission.csv\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.1 Train"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_files = glob.glob(os.path.join(train_dir, \"**/*.txt\"), recursive=True)\nprint(\"Number of files in Train: \", len(train_files))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay now let's see what lies inside a txt file? Feel free to click on `output` button and have a look at the content of a txt file."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"with open(train_files[0], \"r\") as fh:\n for line in fh.readlines():\n print(line)\n fh.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Read this section, if you have read section 3.1 and section 2. Else, read them first.\n\nNow, let us reason together. What do we have in the training folder?\n\n> Each \\*.txt file contains the path trace data, collected by various sensors. And this data corresponds to an indoor path between position p_1 and p_2 walked by a site-surveyor.\n\n\nBut what trace data? Let's have a look. Well, if you have already inspected the relevant code block output, here is what it has to offer:\n\n- startTime: Probably the time the person started navigating\n- SiteId: \n- SiteName: \n- FloorId: \n- FloorName: \n- Brand: \n- Model: \n- AndroidName: \n- APILevel: \n- type * X(where X can be number of sensors in the device): \n- VersionName:\n- VersionCode: \nand `THE TEN` mentioned below."},{"metadata":{},"cell_type":"markdown","source":"What we need, we will see, but ->"},{"metadata":{},"cell_type":"markdown","source":"## What the organizers have to offer:\n\n\n- The first column is Unix Time in millisecond. In specific, we use SensorEvent.timestamp for sensor data and system time for WiFi and Bluetooth scans.\n\n- The second column is the data type (ten in total).\n\n - TYPE_ACCELEROMETER\n - TYPE_MAGNETIC_FIELD\n - TYPE_GYROSCOPE\n - TYPE_ROTATION_VECTOR\n - TYPE_MAGNETIC_FIELD_UNCALIBRATED\n - TYPE_GYROSCOPE_UNCALIBRATED\n - TYPE_ACCELEROMETER_UNCALIBRATED\n - TYPE_WIFI\n - TYPE_BEACON\n - TYPE_WAYPOINT: ground truth location labeled by the surveyor\n\n- Data values start from the third column.\n\n* Column 3-5 of TYPE_ACCELEROMETER、TYPE_ACCELEROMETER、TYPE_GYROSCOPE、TYPE_ROTATION_VECTOR are SensorEvent.values[0-2] from the callback function onSensorChanged(). Column 6 is SensorEvent.accuracy.\n\n* Column 3-8 of TYPE_ACCELEROMETER_UNCALIBRATED、TYPE_GYROSCOPE_UNCALIBRATED、TYPE_MAGNETIC_FIELD_UNCALIBRATED are SensorEvent.values[0-5] from the callback function onSensorChanged(). Column 9 is SensorEvent.accuracy.\n\n* Values of TYPE_BEACON are obtained from ScanRecord.getBytes(). The results are decoded based on iBeacon protocol using the code below.\n\n```\nval major = ((scanRecord[startByte + 20].toInt() and 0xff) * 0x100 + (scanRecord[startByte + 21].toInt() and 0xff))\nval minor = ((scanRecord[startByte + 22].toInt() and 0xff) * 0x100 + (scanRecord[startByte + 23].toInt() and 0xff))\nval txPower = scanRecord[startByte + 24]\n```\n\n* Distance in column 8 is calculated as\n\n```\nprivate static double calculateDistance(int txPower, double rssi) {\n if (rssi == 0) {\n return -1.0; // if we cannot determine distance, return -1.\n }\n double ratio = rssi*1.0/txPower;\n if (ratio < 1.0) {\n return Math.pow(ratio,10);\n }\n else {\n double accuracy = (0.89976)*Math.pow(ratio,7.7095) + 0.111;\n return accuracy;\n }\n}\n```"},{"metadata":{},"cell_type":"markdown","source":"Did you read? Well, it's okay if you didn't. I got you covered. But before that, let's read the Evaluation section again and rethink: `What do we need to do at the first place?` Well, clean the data first xD. Just joking, head over to Evaluation section to better understand. BUt remember that, this section will help us find our way in organizing our data and make it model ready."},{"metadata":{},"cell_type":"markdown","source":"Let's find a way to organize these things:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_txt(txt_path):\n # ignore lines starting with # because they contain meta-data sort of thing\n with open(txt_path, 'r') as fh:\n unique_keys = []\n for line in fh.readlines():\n if line.startswith(\"#\"):\n dummy = line.split(\"\\n\")[0].split(\"\\t\")\n unique_keys.extend(list(map(lambda x: '' if x==\"#\" else x, dummy)))\n else:\n pass\n fh.close()\n return unique_keys\n pass\n\nread_txt(train_files[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.2 Test"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_files = glob.glob(os.path.join(test_dir, \"**/*.txt\"), recursive=True)\nprint(\"Number of files in Test: \", len(test_files))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Feel free to click on the `output` button to see what lies inside a test `*.txt` file."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"with open(test_files[0], \"r\") as fh:\n for line in fh.readlines():\n print(line)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"names = ['Time', 'Type'] + ['col'+str(x) for x in range(1,9)]\ndf = pd.read_csv(train_files[0], sep='\\t', comment='#', header=None, names=names)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.3 Metadata"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"floor_images = glob.glob(os.path.join(meta_dir, \"**/*.png\"), recursive=True)\nfloor_info = glob.glob(os.path.join(meta_dir, \"**/floor_info.json\"), recursive=True)\nGeoMaps = glob.glob(os.path.join(meta_dir, \"**/geojson_map.json\"), recursive=True)\n \nprint(\"Number of Floor Images in Meta Data: \", len(floor_images))\nprint(\"Number of Floor Info(in JSON) in Meta Data: \", len(floor_info))\nprint(\"Number of Geo Map (in JSON) in Meta Data: \", len(GeoMaps))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for _ in range(5):\n img = Image.open(floor_images[np.random.randint(0, len(floor_images))])\n display(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3.4 Sample Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(ss)\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References:\n\n- Competition GitHub Page: https://github.com/location-competition/indoor-location-competition-20\n- Competition Site: https://location20.xyz10.com/"},{"metadata":{"_kg_hide-input":true},"cell_type":"markdown","source":"#