{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Overview\n\n[The result dataset is here.](https://www.kaggle.com/kokitanisaka/unified-ds-wifi-and-beacon)<br>\n<br>\nIn this notebook, I show one way to make a dataset, Wi-Fi features and Beacon features. <br>\nAnd also, I tried to utilize timegap of Wi-Fi and Beacon from the nearest waypoints. <br>\n<br>\nThe fundamental idea of this dataset is, make samples based on waypoints. <br>\nSo the number of samples is same as number of waypoints. <br>\nWhich is much less than [this dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\n<br>\nWe can have similar result only with Wi-Fi features in [this dataset](https://www.kaggle.com/kokitanisaka/unified-ds-wifi-and-beacon) as [this dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\nAs it has less samples, the training speed is much faster.<br>\n<br>\nWith beacon features, I wasn't able to achieve a better result. <br>\nSo if you are interested in it, feel free to do some experiments. <br>\n\n## Attention\nNot all the samples can have beacon features. Because in some paths, there's no beacon signals are observed.<br>\nwe need to take it into account when we train a model. <br>\n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2021-03-12T23:31:59.320227Z","iopub.status.busy":"2021-03-12T23:31:59.319492Z","iopub.status.idle":"2021-03-12T23:31:59.336638Z","shell.execute_reply":"2021-03-12T23:31:59.335911Z"},"papermill":{"duration":0.030327,"end_time":"2021-03-12T23:31:59.336792","exception":false,"start_time":"2021-03-12T23:31:59.306465","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport glob\nimport re\nimport types\ndef imports():\n    for name, val in globals().items():\n        # module imports\n        if isinstance(val, types.ModuleType):\n            yield name, val\n        # functions / callables\n        if hasattr(val, '__call__'):\n            yield name, val\nnp.seterr(divide='ignore', invalid='ignore')\nnoglobal = lambda fn: types.FunctionType(fn.__code__, dict(imports()))\nimport multiprocessing\nfrom multiprocessing import Pool","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:31:59.360742Z","iopub.status.busy":"2021-03-12T23:31:59.360111Z","iopub.status.idle":"2021-03-12T23:31:59.364569Z","shell.execute_reply":"2021-03-12T23:31:59.363759Z"},"papermill":{"duration":0.017673,"end_time":"2021-03-12T23:31:59.364717","exception":false,"start_time":"2021-03-12T23:31:59.347044","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"num_cores = multiprocessing.cpu_count()\n\nbase_path = '/kaggle'","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:31:59.391818Z","iopub.status.busy":"2021-03-12T23:31:59.390963Z","iopub.status.idle":"2021-03-12T23:32:01.880409Z","shell.execute_reply":"2021-03-12T23:32:01.879733Z"},"papermill":{"duration":2.505351,"end_time":"2021-03-12T23:32:01.880558","exception":false,"start_time":"2021-03-12T23:31:59.375207","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# get target buildings\nsample_submission = pd.read_csv(f'{base_path}/input/indoor-location-navigation/sample_submission.csv')\nsample_submission = sample_submission[\"site_path_timestamp\"].apply(lambda x: pd.Series(x.split(\"_\")))\nsample_submission.columns = ['site', 'path', 'timestamp']\ntarget_buildings = sorted(sample_submission['site'].value_counts().index.tolist())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Options\nThese features determin how to take features from the original txt files.<br>\n<br>\nWe take Wi-Fi or beacon features within specific timespan from waypoints. <br>\nI assume that if the time gap is too much, Wi-Fi signals or beacon signals are not trustworthy. <br>\nI set it to 3000ms, but you can try other numbers. <br>\n<br>\nAnd we can determin how many signals to take into the result dataset. <br>\nActually beacon doesn't have much samples in the original txt files, or even doesn't have it. <br>\nYou can try other numbers as well. <br>"},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:32:01.907404Z","iopub.status.busy":"2021-03-12T23:32:01.906404Z","iopub.status.idle":"2021-03-12T23:32:01.908721Z","shell.execute_reply":"2021-03-12T23:32:01.909318Z"},"papermill":{"duration":0.018423,"end_time":"2021-03-12T23:32:01.909492","exception":false,"start_time":"2021-03-12T23:32:01.891069","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# options \nNUM_TAKING_BEACONS = 10\nNUM_TAKING_WIFIS = 100\nTIMEGAP_THRESHOLD = 3000 # ms","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:32:01.937973Z","iopub.status.busy":"2021-03-12T23:32:01.937267Z","iopub.status.idle":"2021-03-12T23:32:01.94046Z","shell.execute_reply":"2021-03-12T23:32:01.94096Z"},"papermill":{"duration":0.02112,"end_time":"2021-03-12T23:32:01.941177","exception":false,"start_time":"2021-03-12T23:32:01.920057","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# constants\nFLOOR_DIR = {\"B2\": -2, \"B1\": -1, \"F1\": 0, \"F2\": 1, \"F3\": 2, \"F4\": 3, \"F5\": 4, \"F6\": 5, \"F7\": 6, \"F8\": 7, \"F9\": 8,\n             \"1F\": 0, \"2F\": 1, \"3F\": 2, \"4F\": 3, \"5F\": 4, \"6F\": 5, \"7F\": 6, \"8F\": 7, \"9F\": 8}","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:32:01.973816Z","iopub.status.busy":"2021-03-12T23:32:01.972617Z","iopub.status.idle":"2021-03-12T23:32:01.998185Z","shell.execute_reply":"2021-03-12T23:32:01.99748Z"},"papermill":{"duration":0.046249,"end_time":"2021-03-12T23:32:01.998339","exception":false,"start_time":"2021-03-12T23:32:01.95209","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# utils\n@noglobal\ndef split_into_each_beacons(s):\n    matches = re.finditer(\"TYPE_BEACON\", s)\n    matches_positions = [match.start() for match in matches]\n    split_idx = [0] + [matches_positions[i]-14 for i in range(1, len(matches_positions))] + [len(s)]\n    return [s[split_idx[i]:split_idx[i+1]] for i in range(len(split_idx)-1)]\n\n@noglobal\ndef extract_waypoint_beacon(path_file):\n    TIME = 0\n\n    WAYPOINT_X = 2\n    WAYPOINT_Y = 3\n    BEACON_DISTANCE = 7\n    BEACON_MAC = 8\n    \n    waypoints = []\n    beacons = []\n    wifis = []\n\n    with open(path_file, encoding=\"utf-8\") as f:\n        text = f.readlines()\n        for i, line in enumerate(text):\n            type_count = line.count('TYPE_BEACON')\n            if type_count > 1:\n                lines = split_into_each_beacons(line)\n            else:\n                lines = [line]\n\n            for l in lines:\n                tmp = l.strip().split()\n                \n                if tmp[1] == \"TYPE_WAYPOINT\":\n                    #1578462618392\tTYPE_WAYPOINT\t230.03738\t153.49635\n                    waypoints.append([int(tmp[TIME]), tmp[1], float(tmp[WAYPOINT_X]), float(tmp[WAYPOINT_Y])])\n\n                elif tmp[1] == \"TYPE_WIFI\":\n                    #1578483067644\tTYPE_WIFI\tda39a3ee5e6b4b0d3255bfef95601890afd80709\t2253c6a0d0f7277737aa8e86e0484be805124806\t-51\t2437\t1578483066126\n                    try:\n                        wifis.append([int(tmp[TIME]), tmp[1], tmp[2], tmp[3], \n                                     int(tmp[4]), int(tmp[5]), int(tmp[6]), 0])\n                    except:\n                        print(tmp)\n                        raise\n                    \n                elif tmp[1] == \"TYPE_BEACON\":\n                    #1578462618698\tTYPE_BEACON\tuuid\tmajor\tminor\t-56\t-58\t1.2902861669921697\tmac\t1578462618698, timediff\n                    try:\n                        if len(tmp) >= 10:                       \n                            second_time = tmp[9]\n                        else:\n                            second_time = tmp[TIME]\n                        \n                        beacons.append([int(tmp[TIME]), tmp[1], tmp[2], tmp[3], tmp[4], int(tmp[5]), int(tmp[6]), \n                                        float(tmp[BEACON_DISTANCE]), tmp[BEACON_MAC], second_time, 0])\n                    except:\n                        print(tmp)\n                        raise\n\n    return waypoints, sorted(beacons, key=lambda x: float(x[BEACON_DISTANCE])), wifis\n\n@noglobal\ndef append_timediff(waypoint, beacons, timediffindex=10):\n    TIME = 0\n    \n    to_be_removed = []\n    for i, beacon in enumerate(beacons):\n        try:\n            beacons[i][timediffindex] = abs(waypoint[TIME] - beacon[TIME])\n        except:\n            to_be_removed.append(i)\n            print(f'error:{beacon}')\n            raise\n    for i in to_be_removed:\n        del beacons[i]\n        \n    return beacons\n\n@noglobal\ndef make_item_wifi(target_building, floor_val, path_val, waypoint, wifis, TIMEGAP_THRESHOLD, NUM_TAKING_WIFIS):\n    WAYPOINT_X = 2\n    WAYPOINT_Y = 3\n\n    sorted_by_nearest = [x for x in wifis if x[7] <= TIMEGAP_THRESHOLD]\n    sorted_by_nearest = sorted(sorted_by_nearest, key=lambda x: (abs(x[4]), x[7]))[:NUM_TAKING_WIFIS]\n\n    item = [target_building, floor_val, path_val, waypoint[WAYPOINT_X], waypoint[WAYPOINT_Y]]\n    for beacon in sorted_by_nearest: \n        item.extend([beacon[3],\n                    beacon[4],\n                    beacon[7]])\n\n    if len(sorted_by_nearest) < NUM_TAKING_WIFIS:\n        for i in range(NUM_TAKING_WIFIS-len(sorted_by_nearest)):\n            item.extend(['-', -999, TIMEGAP_THRESHOLD])    \n\n    return item\n\n@noglobal\ndef append_beacon(target_building, floor_val, path_val, item, waypoint, beacons, TIMEGAP_THRESHOLD, NUM_TAKING_BEACONS):\n    BEACON_DISTANCE = 7\n    BEACON_MAC = 8\n    BEACON_TIMEDIFF = 10\n    WAYPOINT_X = 2\n    WAYPOINT_Y = 3\n\n    sorted_by_nearest_beacons = [x for x in beacons if x[BEACON_TIMEDIFF] <= TIMEGAP_THRESHOLD]\n    sorted_by_nearest_beacons = sorted(sorted_by_nearest_beacons, key=lambda x: x[BEACON_DISTANCE])[:NUM_TAKING_BEACONS]\n\n    #item = [target_building, floor_val, path_val, waypoint[WAYPOINT_X], waypoint[WAYPOINT_Y]]\n    for beacon in sorted_by_nearest_beacons: # select from the nearest beacons\n        item.extend([beacon[BEACON_MAC],\n                    beacon[BEACON_DISTANCE],\n                    beacon[BEACON_TIMEDIFF]])\n\n    if len(sorted_by_nearest_beacons) < NUM_TAKING_BEACONS:\n        for i in range(NUM_TAKING_BEACONS-len(sorted_by_nearest_beacons)):\n            item.extend(['-', -99, TIMEGAP_THRESHOLD])\n\n    return item\n\n@noglobal\ndef yield_columns(NUM_TAKING_WIFIS, NUM_TAKING_BEACONS):\n    columns = []\n    for i in range(NUM_TAKING_WIFIS):\n        columns.append(f'wifi_bssid_{i}')\n        columns.append(f'wifi_rssi_{i}')\n        columns.append(f'wifi_timegap_{i}')\n\n    for i in range(NUM_TAKING_BEACONS):\n        columns.append(f'beacon_macaddress_{i}')\n        columns.append(f'beacon_distance_{i}')\n        columns.append(f'beacon_timegap_{i}')\n\n    return columns","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:32:02.029932Z","iopub.status.busy":"2021-03-12T23:32:02.029255Z","iopub.status.idle":"2021-03-12T23:32:02.03213Z","shell.execute_reply":"2021-03-12T23:32:02.032606Z"},"papermill":{"duration":0.023655,"end_time":"2021-03-12T23:32:02.032787","exception":false,"start_time":"2021-03-12T23:32:02.009132","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def create_data_per_building(target_building):\n    floors = sorted(glob.glob(f'{base_path}/input/indoor-location-navigation/train/{target_building}/*'))\n    \n    items = []\n    \n    for floor in floors:\n        print(floor)\n        \n        floor_val = floor.split('/')[-1]\n        floor_val = FLOOR_DIR[floor_val]\n        \n        paths = sorted(glob.glob(f'{floor}/*.txt'))\n        \n        for path_file in paths:\n            path_val = path_file.split('/')[-1].replace('.txt', '')\n            \n            waypoints, beacons, wifis = extract_waypoint_beacon(path_file)\n\n            for waypoint in waypoints:\n                wifis = append_timediff(waypoint, wifis, 7)\n                beacons = append_timediff(waypoint, beacons, 10)\n                \n                item = make_item_wifi(target_building, floor_val, path_val, waypoint, wifis, TIMEGAP_THRESHOLD, NUM_TAKING_WIFIS)\n                item = append_beacon(target_building, floor_val, path_val, item, waypoint, beacons, TIMEGAP_THRESHOLD, NUM_TAKING_BEACONS)\n            \n                items.append(item)\n                \n                \n    items = pd.DataFrame(items, columns=['site', 'floor', 'path', 'x', 'y'] + yield_columns(NUM_TAKING_WIFIS, NUM_TAKING_BEACONS))\n    \n    items.to_csv(f'{target_building}_train.csv')","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:32:02.121068Z","iopub.status.busy":"2021-03-12T23:32:02.119885Z","iopub.status.idle":"2021-03-12T23:40:33.049491Z","shell.execute_reply":"2021-03-12T23:40:33.050745Z"},"papermill":{"duration":510.94713,"end_time":"2021-03-12T23:40:33.051357","exception":false,"start_time":"2021-03-12T23:32:02.104227","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# make files for train set\nwith Pool(num_cores) as pool:\n    pool.map(create_data_per_building, [t for t in target_buildings])  ","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2021-03-12T23:40:33.167833Z","iopub.status.busy":"2021-03-12T23:40:33.166763Z","iopub.status.idle":"2021-03-12T23:44:20.462037Z","shell.execute_reply":"2021-03-12T23:44:20.461248Z"},"papermill":{"duration":227.354935,"end_time":"2021-03-12T23:44:20.462239","exception":false,"start_time":"2021-03-12T23:40:33.107304","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# make file for test set\n\npaths = sorted(glob.glob(f'{base_path}/input/indoor-location-navigation/test/*'))\n\nitems = []\n\nfor i, path_file in enumerate(paths):\n    path_val = path_file.split('/')[-1].replace('.txt', '')\n\n    print(f'{i}:{path_file}')\n    \n    _, beacons, wifis = extract_waypoint_beacon(path_file)\n\n    targets = sample_submission[sample_submission['path'] == path_val]\n    targets['timestamp'] = targets['timestamp'].astype(int)\n    targets.loc[:,'type'] = 'TYPE_WAYPOINT'\n    targets.loc[:,'x'] = 0\n    targets.loc[:,'y'] = 0\n    waypoints_to_predict = targets[['timestamp', 'type', 'x', 'y']].values.tolist()\n\n    target_building = targets.iloc[0, 0]\n    \n    for waypoint in waypoints_to_predict:\n        wifis = append_timediff(waypoint, wifis, 7)\n        beacons = append_timediff(waypoint, beacons, 10)\n\n        timed = [str(waypoint[0]).zfill(13)]\n        item = make_item_wifi(target_building, 0, path_val, waypoint, wifis, TIMEGAP_THRESHOLD, NUM_TAKING_WIFIS)\n        timed.extend(item)\n        timed = append_beacon(target_building, 0, path_val, timed, waypoint, beacons, TIMEGAP_THRESHOLD, NUM_TAKING_BEACONS)\n        \n        items.append(timed)\n\nitems = pd.DataFrame(items, columns=['timestamp', 'site', 'floor', 'path', 'x', 'y'] + yield_columns(NUM_TAKING_WIFIS, NUM_TAKING_BEACONS))\n\nitems.to_csv(f'test.csv')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}