{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Indoor Location & Navigation  \n### Identify the position of a smartphone in a shopping mall\n\n![logo](https://storage.googleapis.com/kaggle-competitions/kaggle/22559/logos/header.png?t=2020-09-30-17-40-59)"},{"metadata":{},"cell_type":"markdown","source":"\n### Trace File Format（*.txt）(Taken from Competition Github repo)\n\nThe first column is Unix Time in millisecond. In specific, we use SensorEvent.timestamp for sensor data and system time for WiFi and Bluetooth scans. \n\nThe second column is the data type (ten in total).\n* TYPE_ACCELEROMETER\n* TYPE_MAGNETIC_FIELD\n* TYPE_GYROSCOPE\n* TYPE_ROTATION_VECTOR\n* TYPE_MAGNETIC_FIELD_UNCALIBRATED\n* TYPE_GYROSCOPE_UNCALIBRATED\n* TYPE_ACCELEROMETER_UNCALIBRATED\n* TYPE_WIFI\n* TYPE_BEACON\n* TYPE_WAYPOINT: ground truth location labeled by the surveyor\n\nData values start from the third column. \n\n| Time | Data Type                                           | Value                                  |            |       |           |              |            |            |                                 |\n|----------------------|-----------------------------------------------------|------------------------------------------|-------------------|--------------|------------------|---------------------|-------------------|-------------------|----------------------------------------|\n| 1574659531598        | TYPE\\_WAYPOINT                                      | 196\\.41757                               | 117\\.84907        |              |                  |                     |                   |                   |                                        |\n|                      | Location surveyor labeled on the map       | Coordinate x (meter)                             | Coordiante y (meter)     |              |                  |                     |                   |                   |                                        |\n|                      |                                                     |                                          |                   |              |                  |                     |                   |                   |                                        |\n| 1574659531695        | TYPE\\_ACCELEROMETER                                 | \\-1\\.7085724                             | \\-0\\.274765       | 16\\.657166   | 2                |                     |                   |                   |                                        |\n|                      | Android Sensor\\.TYPE\\_ACCELEROMETER                 | X axis                                   | Y axis            | Z axis       | accuracy         |                     |                   |                   |                                        |\n| 1574659531695        | TYPE\\_GYROSCOPE                                     | \\-0\\.3021698                             | 0\\.2773285        | 0\\.107543945 | 3                |                     |                   |                   |                                        |\n|                      | Android Sensor\\.TYPE\\_GYROSCOPE                     | X axis                                   | Y axis            | Z axis       | accuracy         |                     |                   |                   |                                        |\n| 1574659531695        | TYPE\\_MAGNETIC\\_FIELD                               | 20\\.181274                               | 16\\.209412        | \\-32\\.22046  | 3                |                     |                   |                   |                                        |\n|                      | Android Sensor\\.TYPE\\_MAGNETIC\\_FIELD               | X axis                                   | Y axis            | Z axis       | accuracy         |                     |                   |                   |                                        |\n| 1574659531695        | TYPE\\_ROTATION\\_VECTOR                              | \\-0\\.00855688                            | 0\\.051367603      | 0\\.362504    | 3                |                     |                   |                   |                                        |\n|                      | Android Sensor\\.TYPE\\_ROTATION\\_VECTOR              | X axis                                   | Y axis            | Z axis       | accuracy         |                     |                   |                   |                                        |\n|                      |                                                     |                                          |                   |              |                  |                     |                   |                   |                                        |\n| 1574659531695        | TYPE\\_ACCELEROMETER\\_UNCALIBRATED                   | \\-1\\.7085724                             | \\-0\\.274765       | 16\\.657166   | 0\\.0             | 0\\.0                | 0\\.0              | 3                 |                                        |\n|                      | Android Sensor\\.TYPE\\_ACCELEROMETER\\_UNCALIBRATED   | X axis                                   | Y axis            | Z axis       | X axis           | Y axis              | Z axis            | accuracy          |                                        |\n| 1574659531695        | TYPE\\_GYROSCOPE\\_UNCALIBRATED                       | \\-0\\.42333984                            | 0\\.20202637       | 0\\.09623718  | \\-7\\.9345703E\\-4 | 3\\.2043457E\\-4      | 4\\.119873E\\-4     | 3                 |                                        |\n|                      | Android Sensor\\.TYPE\\_GYROSCOPE\\_UNCALIBRATED       | X axis                                   | Y axis            | Z axis       | X axis           | Y axis              | Z axis            | accuracy          |                                        |\n| 1574659531695        | TYPE\\_MAGNETIC\\_FIELD\\_UNCALIBRATED                 | \\-29\\.830933                             | \\-26\\.36261       | \\-300\\.3006  | \\-50\\.012207     | \\-42\\.57202         | \\-268\\.08014      | 3                 |                                        |\n|                      | Android Sensor\\.TYPE\\_MAGNETIC\\_FIELD\\_UNCALIBRATED | X axis                                   | Y axis            | Z axis       | X axis           | Y axis              | Z axis            | accuracy          |                                        |\n|                      |                                                     |                                          |                   |              |                  |                     |                   |                   |                                        |\n| 1574659533190        | TYPE\\_WIFI                                          | intime\\_free                             | 0e:74:9c:a7:b2:e4 | \\-43         | 5805             | 1574659532305       |                   |                   |                                        |\n|                      | Wi\\-Fi data                                         | ssid                                     | bssid             | RSSI         | frequency        | last seen timestamp |                   |                   |                                        |\n|                      |                                                     |                                          |                   |              |                  |                     |                   |                   |                                        |\n| 1574659532751        | TYPE\\_BEACON                                        | FDA50693\\-A4E2\\-4FB1\\-AFCF\\-C6EB07647825 | 10073             | 61418        | \\-65             | \\-82                | 5\\.50634293288929 | 6B:11:4C:D1:29:F2 | 1574659532751                          |\n|                      | iBeacon data                                        | UUID                                     | MajorID           | MinorID      | Tx Power         | RSSI                | Distance          | MAC Address       | same with Unix time, padding data |\n\n\n\n\n\n\nColumn 3-5 of TYPE_ACCELEROMETER、TYPE_ACCELEROMETER、TYPE_GYROSCOPE、TYPE_ROTATION_VECTOR are SensorEvent.values[0-2] from the callback function onSensorChanged(). Column 6 is SensorEvent.accuracy.\n\nColumn 3-8 of TYPE_ACCELEROMETER_UNCALIBRATED、TYPE_GYROSCOPE_UNCALIBRATED、TYPE_MAGNETIC_FIELD_UNCALIBRATED are SensorEvent.values[0-5] from the callback function onSensorChanged(). Column 9 is SensorEvent.accuracy.\n\nValues of TYPE_BEACON are obtained from ScanRecord.getBytes(). The results are decoded based on iBeacon protocol using the code below. \n```\nval major = ((scanRecord[startByte + 20].toInt() and 0xff) * 0x100 + (scanRecord[startByte + 21].toInt() and 0xff))\nval minor = ((scanRecord[startByte + 22].toInt() and 0xff) * 0x100 + (scanRecord[startByte + 23].toInt() and 0xff))\nval txPower = scanRecord[startByte + 24]\n```\nDistance in column 8 is calculated as \n```\nprivate static double calculateDistance(int txPower, double rssi) {\n  if (rssi == 0) {\n    return -1.0; // if we cannot determine distance, return -1.\n  }\n  double ratio = rssi*1.0/txPower;\n  if (ratio < 1.0) {\n    return Math.pow(ratio,10);\n  }\n  else {\n    double accuracy =  (0.89976)*Math.pow(ratio,7.7095) + 0.111;\n    return accuracy;\n  }\n}\n```\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ROOT = \"../input/indoor-location-navigation/\"\nos.listdir(ROOT)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(ROOT+\"train/\")[:3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(os.listdir(ROOT+\"train/\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(ROOT+\"train/\"+\"5cdbc652853bc856e89a8694\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/B1\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!head -n 40 {ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/B1/5d0c23f0c99c56000836d727.txt\"}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!tail -n 40 {ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/B1/5d0c23f0c99c56000836d727.txt\"}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(ROOT+\"test/\")[:3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(os.listdir(ROOT+\"test/\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!head -n 40 {ROOT+\"test/\"+\"52ad8c760ff9978d0949deed.txt\"}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!tail -n 40 {ROOT+\"test/\"+\"52ad8c760ff9978d0949deed.txt\"}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(ROOT+\"sample_submission.csv\")\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['site'] = sub['site_path_timestamp'].apply(lambda x: x.split(\"_\")[0])\nsub['path'] = sub['site_path_timestamp'].apply(lambda x: x.split(\"_\")[1])\nsub['timestamp'] = sub['site_path_timestamp'].apply(lambda x: x.split(\"_\")[2])\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub[sub[\"site\"] == '5d2709d403f801723c32bd39']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub[sub[\"path\"] == '52ad8c760ff9978d0949deed']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list(sub[\"site\"].unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert len(os.listdir(ROOT+\"test/\")) == len((sub['site_path_timestamp'].apply(lambda x: x[:49])).unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set(sub['site'].unique()).issubset(set(os.listdir(ROOT+\"train/\")))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"set([i.split(\".\")[0] for i in os.listdir(ROOT+\"test/\")]) == set(sub['path'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from dataclasses import dataclass\n\nimport numpy as np\n\n\n@dataclass\nclass ReadData:\n    acce: np.ndarray\n    acce_uncali: np.ndarray\n    gyro: np.ndarray\n    gyro_uncali: np.ndarray\n    magn: np.ndarray\n    magn_uncali: np.ndarray\n    ahrs: np.ndarray\n    wifi: np.ndarray\n    ibeacon: np.ndarray\n    waypoint: np.ndarray\n\n\ndef read_data_file(data_filename):\n    acce = []\n    acce_uncali = []\n    gyro = []\n    gyro_uncali = []\n    magn = []\n    magn_uncali = []\n    ahrs = []\n    wifi = []\n    ibeacon = []\n    waypoint = []\n\n    with open(data_filename, 'r', encoding='utf-8') as file:\n        lines = file.readlines()\n\n    for line_data in lines:\n        line_data = line_data.strip()\n        if not line_data or line_data[0] == '#':\n            continue\n\n        line_data = line_data.split('\\t')\n\n        if line_data[1] == 'TYPE_ACCELEROMETER':\n            acce.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ACCELEROMETER_UNCALIBRATED':\n            acce_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE':\n            gyro.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_GYROSCOPE_UNCALIBRATED':\n            gyro_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD':\n            magn.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_MAGNETIC_FIELD_UNCALIBRATED':\n            magn_uncali.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_ROTATION_VECTOR':\n            ahrs.append([int(line_data[0]), float(line_data[2]), float(line_data[3]), float(line_data[4])])\n            continue\n\n        if line_data[1] == 'TYPE_WIFI':\n            sys_ts = line_data[0]\n            ssid = line_data[2]\n            bssid = line_data[3]\n            rssi = line_data[4]\n            lastseen_ts = line_data[6]\n            wifi_data = [sys_ts, ssid, bssid, rssi, lastseen_ts]\n            wifi.append(wifi_data)\n            continue\n\n        if line_data[1] == 'TYPE_BEACON':\n            ts = line_data[0]\n            uuid = line_data[2]\n            major = line_data[3]\n            minor = line_data[4]\n            rssi = line_data[6]\n            ibeacon_data = [ts, '_'.join([uuid, major, minor]), rssi]\n            ibeacon.append(ibeacon_data)\n            continue\n\n        if line_data[1] == 'TYPE_WAYPOINT':\n            waypoint.append([int(line_data[0]), float(line_data[2]), float(line_data[3])])\n\n    acce = np.array(acce)\n    acce_uncali = np.array(acce_uncali)\n    gyro = np.array(gyro)\n    gyro_uncali = np.array(gyro_uncali)\n    magn = np.array(magn)\n    magn_uncali = np.array(magn_uncali)\n    ahrs = np.array(ahrs)\n    wifi = np.array(wifi)\n    ibeacon = np.array(ibeacon)\n    waypoint = np.array(waypoint)\n\n    return ReadData(acce, acce_uncali, gyro, gyro_uncali, magn, magn_uncali, ahrs, wifi, ibeacon, waypoint)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = read_data_file(ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/B1/5d0c23f0c99c56000836d727.txt\")\nprint(f\"acceleration data shape {data.acce.shape}\")\nprint(f\"gyro data shape {data.gyro.shape}\")\nprint(f\"gyro_uncali data shape {data.gyro_uncali.shape}\")\nprint(f\"magn data shape {data.magn.shape}\")\nprint(f\"magn_uncali data shape {data.magn_uncali.shape}\")\nprint(f\"ahrs data shape {data.ahrs.shape}\")\nprint(f\"wifi data shape {data.wifi.shape}\")\nprint(f\"ibeacon data shape {data.ibeacon.shape}\")\nprint(f\"waypoint data shape {data.waypoint.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.acce[:4, :]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.waypoint","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = read_data_file(ROOT+\"test/\"+\"52ad8c760ff9978d0949deed.txt\")\nprint(f\"acceleration data shape {data.acce.shape}\")\nprint(f\"gyro data shape {data.gyro.shape}\")\nprint(f\"gyro_uncali data shape {data.gyro_uncali.shape}\")\nprint(f\"magn data shape {data.magn.shape}\")\nprint(f\"magn_uncali data shape {data.magn_uncali.shape}\")\nprint(f\"ahrs data shape {data.ahrs.shape}\")\nprint(f\"wifi data shape {data.wifi.shape}\")\nprint(f\"ibeacon data shape {data.ibeacon.shape}\")\nprint(f\"waypoint data shape {data.waypoint.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.acce[:4, :]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = read_data_file(ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/B1/5d0c23f0c99c56000836d727.txt\")\nprint(f\"acceleration data shape {data.acce.shape}\")\nprint(f\"gyro data shape {data.gyro.shape}\")\nprint(f\"gyro_uncali data shape {data.gyro_uncali.shape}\")\nprint(f\"magn data shape {data.magn.shape}\")\nprint(f\"magn_uncali data shape {data.magn_uncali.shape}\")\nprint(f\"ahrs data shape {data.ahrs.shape}\")\nprint(f\"wifi data shape {data.wifi.shape}\")\nprint(f\"ibeacon data shape {data.ibeacon.shape}\")\nprint(f\"waypoint data shape {data.waypoint.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Time stamps from one perticular site"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm.auto import tqdm\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths = glob.glob(ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/*/*.txt\")\nstamps = []\nfor i in tqdm(paths, leave=False):\n    data = read_data_file(i)\n    stamps.append(data.acce[:, 0])\n\nfig, axes = plt.subplots(1,1, figsize=(10, 3))\nfor i, j in zip(stamps, paths):\n    plt.plot(i, [1 for _ in range(len(i))])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def getFloorNo(x):\n    if 'F' in x:\n        return int(x.replace(\"F\", \"\")) - 1\n    elif 'B' in x:\n        return -1 * int(x.replace(\"B\", \"\"))\n    else:\n        raise Exception(\"Invalid Floor number\") ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths = glob.glob(ROOT+\"train/\"+\"5cdbc652853bc856e89a8694/*/*.txt\")\nstamps = []\nfor i in tqdm(paths, leave=False):\n    data = read_data_file(i)\n    stamps.append(data.acce[:, 0])\n\nfig, axes = plt.subplots(1,1, figsize=(10, 3))\nfor i, j in zip(stamps, paths):\n    floor = getFloorNo(j.split(\"/\")[-2])\n    plt.plot(i, [floor for _ in range(len(i))])\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths = glob.glob(ROOT+\"train/\"+\"5da138764db8ce0c98bcaa46/*/*.txt\")\nstamps = []\nfor i in tqdm(paths, leave=False):\n    data = read_data_file(i)\n    stamps.append(data.acce[:, 0])\n\nfig, axes = plt.subplots(1,1, figsize=(10, 3))\nfor i, j in zip(stamps, paths):\n    floor = getFloorNo(j.split(\"/\")[-2])\n    plt.plot(i, [floor for _ in range(len(i))])\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths = [ROOT+\"test/\"+i+\".txt\" for i in sub[sub[\"site\"] == '5d2709d403f801723c32bd39'].path.unique()]\nstamps = []\nfor i in tqdm(paths, leave=False):\n    data = read_data_file(i)\n    stamps.append(data.acce[:, 0])\n\nfig, axes = plt.subplots(1,1, figsize=(10, 3))\nfor i, j in zip(stamps, paths):\n    plt.plot(i, [1 for _ in range(len(i))])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths = [ROOT+\"test/\"+i+\".txt\" for i in sub[sub[\"site\"] == '5da138314db8ce0c98bbf3a0'].path.unique()]\nstamps = []\nfor i in tqdm(paths, leave=False):\n    data = read_data_file(i)\n    stamps.append(data.acce[:, 0])\n\nfig, axes = plt.subplots(1,1, figsize=(10, 3))\nfor i, j in zip(stamps, paths):\n    plt.plot(i, [1 for _ in range(len(i))])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the notebook is still WIP but\n### do upvote if it helped :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}