{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Disclaimer\n* Credits go to https://www.kaggle.com/museas for revisiting the cost minimisation solution. \n* I came across this method from https://www.kaggle.com/mehrankazeminia/1-3-indoor-navigation-cost-minimization-floor, which I thank here for their contribution.\n* The submission file used was from the best public ensemble, published in this notebook: https://www.kaggle.com/saurabhbagchi/ensembling-best-performing-notebooksCredits go to https://www.kaggle.com/museas for revisiting the cost minimisation solution. I came across this method from https://www.kaggle.com/mehrankazeminia/1-3-indoor-navigation-cost-minimization-floor, which I thank here for their contribution","metadata":{"_uuid":"9520cd14-356b-40d7-a628-6cc5aa703e62","_cell_guid":"aa4472c2-23a1-4619-ac60-784a22fc8a04","trusted":true}},{"cell_type":"markdown","source":"### Solution follows","metadata":{"_uuid":"ec59acf2-f355-4fef-a3f3-7257cc9bcf30","_cell_guid":"5c27ec0d-3099-4ce1-a323-59ac57696328","trusted":true}},{"cell_type":"markdown","source":"\n\n**The step estimation module provided by the host is great, but sometimes it points to strange positions. That of the host uses TYPE_ROTATION_VECTOR. Therefore, by using it in combination with the estimation using TYPE_MAGNETIC_FIELD, the walking direction can be brought closer to the accurate one.**\n\n\n\nScore improved 0.018 in this sub\n\n4.545 →　4.527","metadata":{"_uuid":"882b44c8-9443-46f4-baaa-bdb95a13b4f6","_cell_guid":"234b5755-d893-450f-a0bf-2967442fb455","trusted":true}},{"cell_type":"markdown","source":"This notebook demonstrates a post-processing strategy for the\n[Indoor Location & Navigation](https://www.kaggle.com/c/indoor-location-navigation)\ncompetition.\n\nTo combine machine learning (wifi features) predictions with sensor data (acceleration, attitude heading),\nI defined cost function as follows,\n$$\nL(X_{1:N}) = \\sum_{i=1}^{N} \\alpha_i \\| X_i - \\hat{X}_i \\|^2 + \\sum_{i=1}^{N-1} \\beta_i \\| (X_{i+1} - X_{i}) - \\Delta \\hat{X}_i \\|^2\n$$\nwhere $\\hat{X}_i$ is absolute position predicted by machine learning and $\\Delta \\hat{X}_i$ is relative position predicted by sensor data.\n\nSince the cost function is quadratic, the optimal $X$ is solved by linear equation $Q X = c$\n, where $Q$ and $c$ are derived from above cost function.\nBecause the matrix $Q$ is tridiagonal,\neach machine learning prediction is corrected by *all* machine learning predictions and sensor data.\n\nThe optimal hyperparameters ($\\alpha$ and $\\beta$) can be estimated by expected error of machine learning and sensor data,\nor just tuned by public score.","metadata":{"_uuid":"384223af-98c9-473b-a6cc-5cae9a8ee167","_cell_guid":"3dc6b183-1df8-45e5-8c79-0f6cba44be2b","trusted":true}},{"cell_type":"markdown","source":"## References\n+ [Simple 99% Accurate Floor Model](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model)\n+ [Indoor Location Competition 2.0 (Sample Data and Code)](https://github.com/location-competition/indoor-location-competition-20)","metadata":{"_uuid":"d637e3da-97c8-4dfe-9fc3-61c0cf1f6ef2","_cell_guid":"d6ba9c14-1c2d-4339-a1d4-3519d5da35ed","trusted":true}},{"cell_type":"code","source":"!git clone --depth 1 https://github.com/location-competition/indoor-location-competition-20 indoor_location_competition_20\n!rm -rf indoor_location_competition_20/data","metadata":{"_uuid":"fdb35183-f6a7-4e9f-9236-dfcc38abe01a","_cell_guid":"d7323aee-5141-4c31-bb14-961bfd4634f6","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import multiprocessing\nimport numpy as np\nimport pandas as pd\nimport scipy.interpolate\nimport scipy.sparse\nfrom tqdm import tqdm\n\nfrom indoor_location_competition_20.io_f import read_data_file\nimport indoor_location_competition_20.compute_f as compute_f","metadata":{"_uuid":"ee21d6de-d5f1-477a-98d9-ea126fda7db9","_cell_guid":"9060571c-0956-4ae7-a4c5-887fc6c76257","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INPUT_PATH = '../input/indoor-location-navigation'","metadata":{"_uuid":"2dee79f1-b378-4feb-a9e3-05d7c13f4aeb","_cell_guid":"4bfdc2e5-eddc-41ba-bdf4-943f5728fe7e","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compute_rel_positions(acce_datas, ahrs_datas):\n    step_timestamps, step_indexs, step_acce_max_mins = compute_f.compute_steps(acce_datas)\n    headings = compute_f.compute_headings(ahrs_datas)\n    stride_lengths = compute_f.compute_stride_length(step_acce_max_mins)\n    step_headings = compute_f.compute_step_heading(step_timestamps, headings)\n    rel_positions = compute_f.compute_rel_positions(stride_lengths, step_headings)\n    return rel_positions","metadata":{"_uuid":"0592ede2-4674-4175-bdac-d5e357220efe","_cell_guid":"64231b64-be27-454e-a4a0-fa5e6570f0cf","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\n\norder = 6\n# fs = 50.0  # sample rate, Hz\n# cutoff = 3\n\nfs = 100\ncutoff = 3.667  # desired cutoff frequency of the filter, Hz\n\nstep_distance = 0.75\nw_height = 1.7\nm_trans = -2","metadata":{"_uuid":"9751fbca-5536-4b99-b388-89071cfeb244","_cell_guid":"1239e246-d76c-456a-831a-98428c1c4d43","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from scipy.signal import butter, lfilter\n\ndef butter_lowpass(cutoff, fs, order=5):\n    nyq = 0.5 * fs\n    normal_cutoff = cutoff / nyq\n    b, a = butter(order, normal_cutoff, btype='low', analog=False)\n    return b, a\n\ndef butter_lowpass_filter(data, cutoff, fs, order=5):\n    b, a = butter_lowpass(cutoff, fs, order=order)\n    y = lfilter(b, a, data)\n    return y","metadata":{"_uuid":"d8bc2188-3e20-4b05-ace0-f17362334c9f","_cell_guid":"2bd6d650-89a4-4e5a-aa00-398f5d27834f","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def peak_accel_threshold(data, timestamps, threshold):\n    d_acc = []\n    last_state = 'below'\n    crest_troughs = 0\n    crossings = []\n\n    for i, datum in enumerate(data):\n        \n        current_state = last_state\n        if datum < threshold:\n            current_state = 'below'\n        elif datum > threshold:\n            current_state = 'above'\n\n        if current_state is not last_state:\n            if current_state is 'above':\n                crossing = [timestamps[i], threshold]\n                crossings.append(crossing)\n            else:\n                crossing = [timestamps[i], threshold]\n                crossings.append(crossing)\n\n            crest_troughs += 1\n        last_state = current_state\n    return np.array(crossings)","metadata":{"_uuid":"04755142-4c19-4d0c-8ae3-de09be10647a","_cell_guid":"b7ff6def-6217-467a-b88a-e2b1136ec67f","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The blending method is to halve the stride length and adopt all steps, the number of steps will be doubled, but here only the movement distance is required**","metadata":{"_uuid":"d4603b86-6b36-4fdd-870c-37b55372fa6a","_cell_guid":"8de8b4e2-5fb3-4bc1-90c7-5c0e2b50ab2f","trusted":true}},{"cell_type":"code","source":"def steps_compute_rel_positions(sample_file):\n    \n    mix_acce = np.sqrt(sample_file.acce[:,1:2]**2 + sample_file.acce[:,2:3]**2 + sample_file.acce[:,3:4]**2)\n    mix_acce = np.concatenate([sample_file.acce[:,0:1], mix_acce], 1)\n    mix_df = pd.DataFrame(mix_acce)\n    mix_df.columns = [\"timestamp\",\"acce\"]\n    \n    filtered = butter_lowpass_filter(mix_df[\"acce\"], cutoff, fs, order)\n\n    threshold = filtered.mean() * 1.1\n    crossings = peak_accel_threshold(filtered, mix_df[\"timestamp\"], threshold)\n\n    step_sum = len(crossings)/2\n    distance = w_height * 0.4 * step_sum\n\n    mag_df = pd.DataFrame(sample_file.magn)\n    mag_df.columns = [\"timestamp\",\"x\",\"y\",\"z\"]\n    \n    acce_df = pd.DataFrame(sample_file.acce)\n    acce_df.columns = [\"timestamp\",\"ax\",\"ay\",\"az\"]\n    \n    mag_df = pd.merge(mag_df,acce_df,on=\"timestamp\")\n    mag_df.dropna()\n    \n    time_di_list = []\n\n    for i in mag_df.iterrows():\n\n        gx,gy,gz = i[1][1],i[1][2],i[1][3]\n        ax,ay,az = i[1][4],i[1][5],i[1][6]\n\n        roll = math.atan2(ay,az)\n        pitch = math.atan2(-1*ax , (ay * math.sin(roll) + az * math.cos(roll)))\n\n        q = m_trans - math.degrees(math.atan2(\n            (gz*math.sin(roll)-gy*math.cos(roll)),(gx*math.cos(pitch) + gy*math.sin(roll)*math.sin(pitch) + gz*math.sin(pitch)*math.cos(roll))\n        )) -90\n        if q <= 0:\n            q += 360\n        time_di_list.append((i[1][0],q))\n\n    d_list = [x[1] for x in time_di_list]\n    \n    steps = []\n    step_time = []\n    di_dict = dict(time_di_list)\n\n    for n,i in enumerate(crossings[:,:1]):\n        if n % 2 == 1:\n            continue\n        direct_now = di_dict[i[0]]\n        dx = math.sin(math.radians(direct_now))\n        dy = math.cos(math.radians(direct_now))\n#         print(int(n/2+1),\"歩目/x:\",dx,\"/y:\",dy,\"/角度：\",direct_now)\n        steps.append((i[0],dx,dy))\n        step_time.append(i[0])\n    \n        step_dtime = np.diff(step_time)/1000\n        step_dtime = step_dtime.tolist()\n        step_dtime.insert(0,5)\n        \n        rel_position = []\n\n        wp_idx = 0\n#         print(\"WP:\",round(sample_file.waypoint[0,1],3),round(sample_file.waypoint[0,2],3),sample_file.waypoint[0,0])\n#         print(\"------------------\")\n        for p,i in enumerate(steps):\n            step_distance = 0\n            if step_dtime[p] >= 1:\n                step_distance = w_height*0.25\n            elif step_dtime[p] >= 0.75:\n                step_distance = w_height*0.3\n            elif step_dtime[p] >= 0.5:\n                step_distance = w_height*0.4\n            elif step_dtime[p] >= 0.35:\n                step_distance = w_height*0.45\n            elif step_dtime[p] >= 0.2:\n                step_distance = w_height*0.5\n            else:\n                step_distance = w_height*0.4\n\n#             step_x += i[1]*step_distance\n#             step_y += i[2]*step_distance\n            \n            rel_position.append([i[0], i[1]*step_distance, i[2]*step_distance])\n#     print(rel_position)\n    \n    return np.array(rel_position)","metadata":{"_uuid":"33c6b0cd-fa90-41fb-94fd-99c879c20e30","_cell_guid":"705ebba0-7fbd-45f7-be95-ab3ad4f728a6","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def correct_path(args):\n    path, path_df = args\n    \n    T_ref  = path_df['timestamp'].values\n    xy_hat = path_df[['x', 'y']].values\n    \n    example = read_data_file(f'{INPUT_PATH}/test/{path}.txt')\n\n    rel_positions1 = compute_rel_positions(example.acce, example.ahrs)\n    rel_positions2 = steps_compute_rel_positions(example)\n    rel1 = rel_positions1.copy()\n    rel2 = rel_positions2.copy()\n    rel1[:,1:] = rel_positions1[:,1:] / 2\n    rel2[:,1:] = rel_positions2[:,1:] / 2\n    rel_positions = np.vstack([rel1,rel2])\n    rel_positions = rel_positions[np.argsort(rel_positions[:, 0])]\n    \n    if T_ref[-1] > rel_positions[-1, 0]:\n        rel_positions = [np.array([[0, 0, 0]]), rel_positions, np.array([[T_ref[-1], 0, 0]])]\n    else:\n        rel_positions = [np.array([[0, 0, 0]]), rel_positions]\n    rel_positions = np.concatenate(rel_positions)\n    \n    T_rel = rel_positions[:, 0]\n    delta_xy_hat = np.diff(scipy.interpolate.interp1d(T_rel, np.cumsum(rel_positions[:, 1:3], axis=0), axis=0)(T_ref), axis=0)\n\n    N = xy_hat.shape[0]\n    delta_t = np.diff(T_ref)\n    alpha = (8.1)**(-2) * np.ones(N)\n    beta  = (0.3 + 0.3 * 1e-3 * delta_t)**(-2)\n    A = scipy.sparse.spdiags(alpha, [0], N, N)\n    B = scipy.sparse.spdiags( beta, [0], N-1, N-1)\n    D = scipy.sparse.spdiags(np.stack([-np.ones(N), np.ones(N)]), [0, 1], N-1, N)\n\n    Q = A + (D.T @ B @ D)\n    c = (A @ xy_hat) + (D.T @ (B @ delta_xy_hat))\n    xy_star = scipy.sparse.linalg.spsolve(Q, c)\n\n    return pd.DataFrame({\n        'site_path_timestamp' : path_df['site_path_timestamp'],\n        'floor' : path_df['floor'],\n        'x' : xy_star[:, 0],\n        'y' : xy_star[:, 1],\n    })","metadata":{"_uuid":"e9cf63a4-c33c-4103-9507-d62e9868c5f6","_cell_guid":"55b5ce1f-6476-4d0a-997f-3a207331e121","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.read_csv('../input/indoor-loc-and-nav-subs/submission_4475.csv')  # ('../input/simple-99-accurate-floor-model/submission.csv')\ntmp = sub['site_path_timestamp'].apply(lambda s : pd.Series(s.split('_')))\nsub['site'] = tmp[0]\nsub['path'] = tmp[1]\nsub['timestamp'] = tmp[2].astype(float)\n\nprocesses = multiprocessing.cpu_count()\nwith multiprocessing.Pool(processes=processes) as pool:\n    dfs = pool.imap_unordered(correct_path, sub.groupby('path'))\n    dfs = tqdm(dfs)\n    dfs = list(dfs)\nsub = pd.concat(dfs).sort_values('site_path_timestamp')\nsub.to_csv('first-submission.csv', index=False)","metadata":{"_uuid":"173bee8f-c57c-4263-979a-40e4cd1e00c9","_cell_guid":"8c64723d-576c-4ac3-a3e8-f50db5bbc2a7","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"80006d97-ca93-4a26-9121-eaf8cfb48a41","_cell_guid":"03c5a806-4227-4f68-aee8-76964097b6a2","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Postprocessing with leaked feature\n\nIn the discussion section, I proposed [the possiility of the leakage caused by raw timestamps](https://www.kaggle.com/c/indoor-location-navigation/discussion/228898).\n\nIn short.\n* The last waypoints of some paths are exactly the same as the first waypoints of other paths. I think this is because they are divided from a single measurement.\n* The raw timestamps have information for the floor. I think this is because the measurements took place in floor by floor manner.\n\nIn this notebook, I demonstrate the existance of the leakage by presenting the results of postprocessing based on the raw timestamps  and not on any sensor information are actually good or even better.\n\nThe algorithm of postprocessing is simple.\n* For x and y of the first waypoint, I use those of the temporally nearest endpoint in training dataset.\n* For x and y of the last waypoint, I use those of the temporally nearest startpoint in training dataset.\n* For floor of all waypoints, I use those of the temporally nearest endpoint and startpoint in training dataset if they match each other.\n\nI have not decided whether I use this leakage. Should I try to create the model valid for real life or just to win this competition? I hope the competition host recreated the test dataset without any leakages.\n\nI use some codes, data, and ideas from following notebooks. Thank you very much.\n* https://www.kaggle.com/kenmatsu4/feature-store-for-indoor-location-navigation\n* https://www.kaggle.com/mehrankazeminia/3-3-g6-indoor-navigation-snap-to-grid\n* https://www.kaggle.com/jiweiliu/fix-the-timestamps-of-test-data-using-dask\n\n[update] I found a bug in calculating the raw time stamp and fix it. The score changed worse a little bit.\n[update2] I found another bug in getting test data function and fix it. Then the score changed better a little bit.","metadata":{"_uuid":"5d72205e-715a-467f-9a62-919cb8077003","_cell_guid":"81bc08a1-8174-4911-a04b-59a8f6d61e8a","trusted":true}},{"cell_type":"code","source":"import json\nimport re\nimport gc\nimport pickle\nimport itertools\nimport pandas as pd\nimport numpy as np\nfrom glob import glob\nfrom datetime import datetime as dt\nfrom pathlib import Path\nfrom tqdm import tqdm\nimport datetime\nts_conv = np.vectorize(datetime.datetime.fromtimestamp) # ut(10 digit) -> date\n\n# pandas settings -----------------------------------------\npd.set_option(\"display.max_colwidth\", 100)\npd.set_option(\"display.max_rows\", None)\npd.set_option(\"display.max_columns\", None)\npd.options.display.float_format = '{:,.5f}'.format\n\n# Graph drawing -------------------------------------------\nimport matplotlib\nfrom matplotlib import font_manager\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\nfrom matplotlib import rc\nfrom matplotlib_venn import venn2, venn2_circles\nfrom matplotlib import animation as ani\nfrom IPython.display import Image\nfrom pylab import imread\n\nplt.rcParams[\"patch.force_edgecolor\"] = True\nfrom IPython.display import display # Allows the use of display() for DataFrames\nimport seaborn as sns\nsns.set(style=\"whitegrid\", palette=\"muted\", color_codes=True)\nsns.set_style(\"whitegrid\", {'grid.linestyle': '--'})\nred = sns.xkcd_rgb[\"light red\"]\ngreen = sns.xkcd_rgb[\"medium green\"]\nblue = sns.xkcd_rgb[\"denim blue\"]\n\n%matplotlib inline\n%config InlineBackend.figure_format='retina'\n\n# ML -------------------------------------------\nfrom sklearn.preprocessing import LabelEncoder\n\n\nimport dill\nfrom collections import defaultdict, OrderedDict\nfrom scipy.spatial import distance","metadata":{"_uuid":"4d54ea6d-c7d6-4ccf-b7eb-bb0b91825794","_cell_guid":"c1313fd3-59d6-4cb2-bbdb-aaad59b7b2c3","collapsed":false,"_kg_hide-input":true,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def unpickle(filename):\n    with open(filename, 'rb') as fo:\n        p = pickle.load(fo)\n    return p\n\ndef to_pickle(filename, obj):\n    with open(filename, 'wb') as f:\n        pickle.dump(obj, f, -1)\n\n\n\nclass FeatureStore():\n    \n    # necessayr to re-check\n    floor_convert = {'1F' :  0, '2F' : 1, '3F' : 2, '4F' : 3, '5F' : 4, \n                     '6F' : 5, '7F' : 6, '8F' : 7, '9F' : 8,\n                     'B'  : -1, 'B1' : -1, 'B2' : -2, 'B3' : -3, \n                     'BF' : -1, 'BM' : -1, \n                     'F1' : 0, 'F2' : 1, 'F3' : 2, 'F4' : 3, 'F5' : 4, \n                     'F6' : 5, 'F7' : 6, 'F8' : 7, 'F9' : 8, 'F10': 9,\n                     'L1' : 0, 'L2' : 1, 'L3' : 2, 'L4' : 3, 'L5' : 4, \n                     'L6' : 5, 'L7' : 6, 'L8' : 7, 'L9' : 8, 'L10': 9, \n                     'L11': 10,\n                     'G'  : 0, 'LG1': 0, 'LG2': 1, 'LM' : 0, 'M'  : 0, \n                     'P1' : 0, 'P2' : 1,}\n    \n    df_types = ['accelerometer',\n                'accelerometer_uncalibrated',\n                'beacon',\n                'gyroscope',\n                'gyroscope_uncalibrated',\n                'magnetic_field',\n                'magnetic_field_uncalibrated',\n                'rotation_vector',\n                'waypoint',\n                'wifi']\n    \n    # https://github.com/location-competition/indoor-location-competition-20\n    df_type_cols = {'accelerometer': [\"timestamp\", \"x\", \"y\", \"z\", \"accuracy\"],\n                'accelerometer_uncalibrated': [\"timestamp\", \"x\", \"y\", \"z\", \n                                               \"x2\", \"y2\", \"z2\", \"accuracy\" ],\n                'beacon': [\"timestamp\", \"uuid\", \"major_id\", \"minor_id\", \"tx_power\", \n                           \"rssi\", \"distance\", \"mac_addr\", \"timestamp2\"],\n                'gyroscope': [\"timestamp\", \"x\", \"y\", \"z\", \"accuracy\"],\n                'gyroscope_uncalibrated': [\"timestamp\", \"x\", \"y\", \"z\", \n                                           \"x2\", \"y2\", \"z2\", \"accuracy\" ],\n                'magnetic_field': [\"timestamp\", \"x\", \"y\", \"z\", \"accuracy\"],\n                'magnetic_field_uncalibrated': [\"timestamp\", \"x\", \"y\", \"z\", \n                                                \"x2\", \"y2\", \"z2\", \"accuracy\" ],\n                'rotation_vector': [\"timestamp\", \"x\", \"y\", \"z\", \"accuracy\"],\n                'waypoint': [\"timestamp\", \"x\", \"y\"],\n                'wifi': [\"timestamp\", \"ssid\", \"bssid\",\"rssi\",\"frequency\",\n                         \"last_seen_timestamp\",]}\n\n    dtype_dict = {}\n    dtype_dict[\"accelerometer\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \"z\":float, \n                                   \"accuracy\":int}\n    dtype_dict[\"accelerometer_uncalibrated\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \n                                                \"z\":float, \"x2\":float, \"y2\":float, \n                                                \"z2\":float, \"accuracy\":int}\n    dtype_dict[\"beacon\"] = {\"timestamp\":int, \"uuid\":str, \"major_id\":str, \n                            \"minor_id\":str, \"tx_power\":int,  \"rssi\":int, \n                            \"distance\":float, \"mac_addr\":str, \"timestamp2\":int}\n    dtype_dict[\"gyroscope\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \"z\":float, \n                               \"accuracy\":int}\n    dtype_dict[\"gyroscope_uncalibrated\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \n                                            \"z\":float, \"x2\":float, \"y2\":float, \n                                            \"z2\":float, \"accuracy\":int}\n    dtype_dict[\"magnetic_field\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \n                                    \"z\":float, \"accuracy\":int}\n    dtype_dict[\"magnetic_field_uncalibrated\"] = {\"timestamp\":int, \"x\":float, \n                                                 \"y\":float, \"z\":float, \"x2\":float, \n                                                 \"y2\":float, \"z2\":float, \"accuracy\":int}\n    dtype_dict[\"rotation_vector\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \n                                     \"z\":float, \"accuracy\":int}\n    dtype_dict[\"waypoint\"] = {\"timestamp\":int, \"x\":float, \"y\":float, \"z\":float}\n    dtype_dict[\"wifi\"] = {\"timestamp\":int, \"ssid\":str, \"bssid\":str,\n                          \"rssi\":int,\"frequency\":int, \"last_seen_timestamp\":int}\n\n    def __init__(self, site_id, floor, path_id, \n                 input_path=\"../input/indoor-location-navigation/\",\n                 save_path=\"../mid\"):\n        self.site_id = site_id.strip()\n        self.floor = floor.strip()\n        self.n_floor = self.floor_convert[self.floor]\n        self.path_id = path_id.strip()\n        \n        self.input_path = input_path\n        assert Path(input_path).exists(), f\"input_path do not exist: {input_path}\"\n        \n        self.save_path = save_path\n        Path(save_path).mkdir(parents=True, exist_ok=True)\n        \n        self.site_info = SiteInfo(site_id=self.site_id, floor=self.floor, input_path=self.input_path)\n        \n    def _flatten(self, l):\n        return list(itertools.chain.from_iterable(l))\n    \n    def multi_line_spliter(self, s):\n        matches = re.finditer(\"TYPE_\", s)\n        matches_positions = [match.start() for match in matches]\n        split_idx = [0] + [matches_positions[i]-14 for i in range(1, len(matches_positions))] + [len(s)]\n        return [s[split_idx[i]:split_idx[i+1]] for i in range(len(split_idx)-1)]\n    \n    def load_df(self, ):\n        path = str(Path(self.input_path)/f\"train/{self.site_id}/{self.floor}/{self.path_id}.txt\")\n        with open(path) as f:\n            data = f.readlines()\n        \n        modified_data = []\n        for s in data:\n            if s.count(\"TYPE_\")>1:\n                lines = self.multi_line_spliter(s)\n                modified_data.extend(lines)\n            else:\n                modified_data.append(s)\n        del data\n        self.meta_info_len = len([d for d in modified_data if d[0]==\"#\"])\n        self.meta_info_df = pd.DataFrame([m.replace(\"\\n\", \"\").split(\":\") \n                                          for m in self._flatten([d.split(\"\\t\") \n                                                                  for d in modified_data if d[0]==\"#\"]) if m!=\"#\"])\n\n        data_df = pd.DataFrame([d.replace(\"\\n\", \"\").split(\"\\t\") for d in modified_data if d[0]!=\"#\"])\n        for dt in self.df_types:\n            # select data type\n            df_s = data_df[data_df[1]==f\"TYPE_{dt.upper()}\"]\n            if len(df_s)==0:\n                setattr(self, dt, pd.DataFrame(columns=self.df_type_cols[dt]))\n            else:\n                # remove empty cols\n                na_info = df_s.isna().sum(axis=0) == len(df_s)\n                df_s = df_s[[i for i in na_info[na_info==False].index if i!=1]].reset_index(drop=True)\n                \n                if len(df_s.columns)!=len(self.df_type_cols[dt]):\n                    df_s.columns = self.df_type_cols[dt][:len(df_s.columns)]\n                else:\n                    df_s.columns = self.df_type_cols[dt]\n            \n                # set dtype          \n                for c in df_s.columns:\n                    df_s[c] = df_s[c].astype(self.dtype_dict[dt][c])\n                                     \n                # set DataFrame to attr\n                setattr(self, dt, df_s)\n    \n    def get_site_info(self, keep_raw=False):\n        self.site_info.get_site_info(keep_raw=keep_raw)\n            \n    def load_all_data(self, keep_raw=False):     \n        self.load_df()\n        self.get_site_info(keep_raw=keep_raw)\n        \n    def __getitem__(self, item):\n        if item in self.df_types:\n            return getattr(self, item)\n        else:\n            return None\n    \n    def save(self, ):\n        # to be implemented\n        pass\n    \n    \nclass SiteInfo():\n    def __init__(self, site_id, floor, input_path=\"../input/indoor-location-navigation/\"):\n        self.site_id = site_id\n        self.floor = floor\n        self.input_path = input_path\n        assert Path(input_path).exists(), f\"input_path do not exist: {input_path}\"\n        \n    def get_site_info(self, keep_raw=False):\n        floor_info_path = f\"{self.input_path}/metadata/{self.site_id}/{self.floor}/floor_info.json\"\n        with open(floor_info_path, \"r\") as f:\n            self.floor_info = json.loads(f.read())\n            self.site_height = self.floor_info[\"map_info\"][\"height\"]\n            self.site_width = self.floor_info[\"map_info\"][\"width\"]\n            if not keep_raw:\n                del self.floor_info\n            \n        geojson_map_path = f\"{self.input_path}/metadata/{self.site_id}/{self.floor}/geojson_map.json\"\n        with open(geojson_map_path, \"r\") as f:\n            self.geojson_map = json.loads(f.read())\n            self.map_type = self.geojson_map[\"type\"]\n            self.features = self.geojson_map[\"features\"]\n            \n            self.floor_coordinates = self.features[0][\"geometry\"][\"coordinates\"]\n            self.store_coordinates = [self.features[i][\"geometry\"][\"coordinates\"] \n                                          for i in range(1, len(self.features))]\n                \n            if not keep_raw:\n                del self.geojson_map\n    \n    def show_site_image(self):\n        path = f\"{self.input_path}/metadata/{self.site_id}/{self.floor}/floor_image.png\"\n        plt.imshow(imread(path), extent=[0, self.site_width, 0, self.site_height])\n\n    def draw_polygon(self, size=8, only_floor=False):\n\n        fig = plt.figure()\n        ax = plt.subplot(111)\n            \n        xmax, xmin, ymax, ymin = self._draw(self.floor_coordinates, ax, calc_minmax=True)\n        if not only_floor:\n            self._draw(self.store_coordinates, ax, fill=True)\n        plt.legend([])\n        \n        xrange = xmax - xmin\n        yrange = ymax - ymin\n        ratio = yrange / xrange\n        \n        self.x_size = size\n        self.y_size = size*ratio\n\n        fig.set_figwidth(size)\n        fig.set_figheight(size*ratio)\n        # plt.show()\n        return ax\n        \n    def _draw(self, coordinates, ax, fill=False, calc_minmax=False):\n        xmax, ymax = -np.inf, -np.inf\n        xmin, ymin = np.inf, np.inf\n        for i in range(len(coordinates)):\n            ndim = np.ndim(coordinates[i])\n            if ndim==2:\n                corrd_df = pd.DataFrame(coordinates[i])\n                if fill:\n                    ax.fill(corrd_df[0], corrd_df[1], alpha=0.7)\n                else:\n                    corrd_df.plot.line(x=0, y=1, style=\"-\", ax=ax)\n                        \n                if calc_minmax:\n                    xmax = max(xmax, corrd_df[0].max())\n                    xmin = min(xmin, corrd_df[0].min())\n\n                    ymax = max(ymax, corrd_df[1].max())\n                    ymin = min(ymin, corrd_df[1].min())\n            elif ndim==3:\n                for j in range(len(coordinates[i])):\n                    corrd_df = pd.DataFrame(coordinates[i][j])\n                    if fill:\n                        ax.fill(corrd_df[0], corrd_df[1], alpha=0.6)\n                    else:\n                        corrd_df.plot.line(x=0, y=1, style=\"-\", ax=ax)\n                        \n                    if calc_minmax:\n                        xmax = max(xmax, corrd_df[0].max())\n                        xmin = min(xmin, corrd_df[0].min())\n\n                        ymax = max(ymax, corrd_df[1].max())\n                        ymin = min(ymin, corrd_df[1].min())\n            else:\n                assert False, f\"ndim of coordinates should be 2 or 3: {ndim}\"\n        if calc_minmax:\n            return xmax, xmin, ymax, ymin\n        else:\n            return None","metadata":{"_uuid":"d7c41a1f-9b55-4734-8d29-ff5b6eb0928d","_cell_guid":"c3e25ec8-56ac-45f2-8ad3-5abe3d2de7fb","collapsed":false,"_kg_hide-input":true,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_meta_data\ntrain_meta = glob(\"../input/indoor-location-navigation/train/*/*/*\")\ntrain_meta_org = pd.DataFrame(train_meta)\ntrain_meta = train_meta_org[0].str.split(\"/\", expand=True)[[4, 5, 6]]\ntrain_meta.columns = [\"site_id\", \"floor\", \"path_id\"]\ntrain_meta[\"path_id\"] = train_meta[\"path_id\"].str.replace(\".txt\", \"\")\ntrain_meta[\"path\"] = train_meta_org[0]\n#train_meta.head()","metadata":{"_uuid":"36243319-85e7-40f0-8045-0a4560382c1f","_cell_guid":"d4083cea-ad72-439d-9102-3ef021a499f5","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pickle_dump_dill(obj, path):\n    with open(path, mode='wb') as f:\n        dill.dump(obj, f)\n\n\ndef pickle_load_dill(path):\n    with open(path, mode='rb') as f:\n        data = dill.load(f)\n        return data","metadata":{"_uuid":"d6101b48-c3cc-4f9f-a763-f210c7509af5","_cell_guid":"780760a3-3d48-4397-9d88-4001947456c8","collapsed":false,"_kg_hide-input":true,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sub = pd.read_csv('../input/indoor-location-navigation/sample_submission.csv')\ntest_sites = sample_sub.site_path_timestamp.apply(lambda x: pd.Series(x.split(\"_\")))[0].unique().tolist()\n\ntest_meta = sample_sub[\"site_path_timestamp\"].apply(\n    lambda x: pd.Series(x.split(\"_\")))\ntest_meta.columns = [\"site_id\", \"path_id\", \"timestamp\"]\ntest_meta=test_meta.drop('timestamp', axis=1)\ntest_meta = test_meta.drop_duplicates(subset=[\"site_id\", \"path_id\"]).reset_index(drop=True)","metadata":{"_uuid":"1cf204d1-d28f-4da9-8e12-18b601b61450","_cell_guid":"f65edb83-5e43-4df7-8f29-9a382b03bba9","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Get first and last waypoints in train dataset","metadata":{"_uuid":"c24617f1-e036-4b33-b132-4eff9c6cb097","_cell_guid":"13f31c39-4118-4d32-aeef-7f6dd23a4018","trusted":true}},{"cell_type":"code","source":"create_train_meta_sub=False\nif create_train_meta_sub:\n    train_meta_sub=train_meta[train_meta['site_id'].isin(test_sites)].reset_index(drop=True)\n    train_meta_sub['start_time']=0\n    train_meta_sub['end_time']=0\n    train_meta_sub['start_wp_time']=0\n    train_meta_sub['start_wp_x']=0\n    train_meta_sub['start_wp_y']=0\n    train_meta_sub['end_wp_time']=0\n    train_meta_sub['end_wp_x']=0\n    train_meta_sub['end_wp_y']=0\n    train_meta_sub['n_floor']=0\n    for i in tqdm(range(len(train_meta_sub))):\n        t = train_meta_sub.iloc[i]\n        n_floor = FeatureStore.floor_convert[t.floor]\n        feature = FeatureStore(\n            site_id=t.site_id, floor=t.floor, path_id=t.path_id)\n        feature.load_all_data() \n        start_time=int(feature.meta_info_df[feature.meta_info_df[0]=='startTime'][1])\n        end_time=int(feature.meta_info_df[feature.meta_info_df[0]=='endTime'][1])\n        train_meta_sub.loc[i,'start_time']=start_time\n        train_meta_sub.loc[i,'start_wp_time']=feature.waypoint.iloc[0]['timestamp']\n        train_meta_sub.loc[i,'start_wp_x']=feature.waypoint.iloc[0]['x']\n        train_meta_sub.loc[i,'start_wp_y']=feature.waypoint.iloc[0]['y']\n        train_meta_sub.loc[i,'end_time']=end_time\n        train_meta_sub.loc[i,'end_wp_time']=feature.waypoint.iloc[-1]['timestamp']\n        train_meta_sub.loc[i,'end_wp_x']=feature.waypoint.iloc[-1]['x']\n        train_meta_sub.loc[i,'end_wp_y']=feature.waypoint.iloc[-1]['y']\n        train_meta_sub.loc[i,'n_floor']=feature.n_floor\n    train_meta_sub.to_csv('train_meta_sub.csv', index=False)\nelse:\n    train_meta_sub = pd.read_csv('../input/indoor-public/train_meta_sub.csv')","metadata":{"_uuid":"31ae60b2-5d6d-44fa-b93e-42f7e7d0a0cc","_cell_guid":"7801beed-bde6-4f3e-86b0-6d7da1d3bcfd","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_meta_sub[train_meta_sub.site_id=='5d2709b303f801723c327472'][['path_id','site_id','n_floor','start_time','start_wp_x','start_wp_y','end_time','end_wp_x','end_wp_y']].sort_values(['site_id','n_floor','start_time'])[:50]","metadata":{"_uuid":"6ca882f9-2742-4a4b-b33a-1739386cdae5","_cell_guid":"e817cf95-7ed0-4973-9e58-e9d768bc9939","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see the last waypoints of some paths are exactly the same as the first waypoints of other paths in training dataset.","metadata":{"_uuid":"b3977ec9-758c-4db2-b187-3d86a3ac9358","_cell_guid":"7dc26299-1365-4ae2-8cfb-14b5a1be8a61","trusted":true}},{"cell_type":"code","source":"import seaborn as sns\nfor test_site in test_sites:\n    plt.figure()\n    plt.title(test_site)\n    sns.boxplot(x='floor', y='start_time', data=train_meta_sub[train_meta_sub.site_id==test_site])","metadata":{"_uuid":"131dfa54-6b44-4802-a549-0bf7b5b1f80a","_cell_guid":"6e480e67-401b-4377-9af7-e943e029c188","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see the raw timestamps have information for the floor in training dataset.","metadata":{"_uuid":"f078b1ec-8ddf-4e56-834b-9091c38da509","_cell_guid":"92d6386c-2df9-4284-96bc-18e96691bc41","trusted":true}},{"cell_type":"code","source":"def read_txt(file):\n    with open(file) as f:\n        txt = f.readlines()\n\n    modified_data = []\n    for s in txt:\n        if s.count(\"TYPE_\") > 1:\n            lines = multi_line_spliter(s)\n            modified_data.extend(lines)\n        else:\n            modified_data.append(s)\n    return modified_data\n\n\ndef _flatten(l):\n    return list(itertools.chain.from_iterable(l))\n\n\ndef get_feature_test(site_id, path_id, input_path, sample_sub):\n    file = f\"{input_path}/test/{path_id}.txt\"\n    content = read_txt(file)\n    data_df = pd.DataFrame([d.replace(\"\\n\", \"\").split(\"\\t\")\n                            for d in content if d[0] != \"#\"])\n    data_dict = OrderedDict()\n    for dt in FeatureStore.df_types:\n        # select data type\n        df_s = data_df[data_df[1] == f\"TYPE_{dt.upper()}\"]\n        if len(df_s) == 0:\n            setattr(data_dict, dt, pd.DataFrame(\n                columns=FeatureStore.df_type_cols[dt]))\n        else:\n            # remove empty cols\n            na_info = df_s.isna().sum(axis=0) == len(df_s)\n            df_s = df_s[[i for i in na_info[na_info ==\n                                            False].index if i != 1]].reset_index(drop=True)\n\n            if len(df_s.columns) != len(FeatureStore.df_type_cols[dt]):\n                df_s.columns = FeatureStore.df_type_cols[dt][:len(\n                    df_s.columns)]\n            else:\n                df_s.columns = FeatureStore.df_type_cols[dt]\n\n            # set dtype\n            for c in df_s.columns:\n                df_s[c] = df_s[c].astype(FeatureStore.dtype_dict[dt][c])\n            setattr(data_dict, dt, df_s)\n    data_dict.meta_info_df = pd.DataFrame([m.replace(\"\\n\", \"\").split(\":\")\n                                           for m in _flatten([d.split(\"\\t\")\n                                                              for d in content if d[0] == \"#\"]) if m != \"#\"])\n    startTime_ind = int(np.where(data_dict.meta_info_df[0] == 'startTime')[0])\n    endTime_ind = int(np.where(data_dict.meta_info_df[0] == 'endTime')[0])\n    data_dict.meta_info_df.loc[startTime_ind,\n                               1] = data_dict.meta_info_df.loc[startTime_ind+1, 0]\n    data_dict.meta_info_df.loc[endTime_ind,\n                               1] = data_dict.meta_info_df.loc[endTime_ind+1, 0]\n\n    data_dict.waypoint['timestamp'] = sample_sub[sample_sub.path_id ==\n                                                 path_id].timestamp.values.astype(int)\n    data_dict.waypoint['x'] = 0\n    data_dict.waypoint['y'] = 0\n    data_dict.n_floor = 0\n    data_dict.site_id = site_id\n    return data_dict","metadata":{"_uuid":"636443cc-f01f-447e-9978-47162fe2e21f","_cell_guid":"2ad952e2-d8f3-4d2e-bf3b-96089d3b4993","collapsed":false,"_kg_hide-input":false,"_kg_hide-output":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Postprocessing based on leaked feature.","metadata":{"_uuid":"7f0e7ce5-e202-463f-961d-218c45ebb0f4","_cell_guid":"004ad2e7-ed79-4a34-9f2f-a6c68b382058","trusted":true}},{"cell_type":"code","source":"def leak_postprocessing(submission_df,train_meta, postprocess_start=True, postprocess_end=True, postprocess_floor=True,start_threshold=5500,end_threshold=5500):\n    out_df=submission_df.copy()\n    out_df[[\"site_id\", \"path_id\", \"timestamp\"]] = out_df[\"site_path_timestamp\"].apply(\n        lambda x: pd.Series(x.split(\"_\")))\n    start_counter = 0\n    end_counter = 0\n    floor_counter = 0\n    input_path='/kaggle/input/indoor-location-navigation/'\n    sample_sub = pd.read_csv(f\"{input_path}/sample_submission.csv\")\n    sample_sub = sample_sub[\"site_path_timestamp\"].apply(\n        lambda x: pd.Series(x.split(\"_\")))\n    sample_sub.columns = [\"site_id\", \"path_id\", \"timestamp\"]\n    out_df_unique=out_df.drop_duplicates(\n    subset=[\"site_id\", \"path_id\"]).reset_index(drop=True)\n    for i in tqdm(range(len(out_df_unique.path_id))):\n        t = out_df_unique.iloc[i]\n        site_id=t.site_id\n        path_id=t.path_id\n        feature = get_feature_test(site_id, path_id, input_path, sample_sub)\n        if feature.meta_info_df[feature.meta_info_df[0] == 'startTime'][1].values == None:\n            start_time = int(np.nanmin([feature.accelerometer.timestamp.min(\n            ), feature.wifi.timestamp.min(), feature.beacon.timestamp.min()]))\n        else:\n            start_time = int(\n                feature.meta_info_df[feature.meta_info_df[0] == 'startTime'][1])\n        if (len(feature.meta_info_df[feature.meta_info_df[0] == 'endTime']) == 0) or (feature.meta_info_df[feature.meta_info_df[0] == 'endTime'][1].values == None):\n            end_time = int(np.nanmax([feature.accelerometer.timestamp.max(\n            ), feature.wifi.timestamp.max(), feature.beacon.timestamp.max()]))\n        else:\n            end_time = int(\n                feature.meta_info_df[feature.meta_info_df[0] == 'endTime'][1])\n        if len(feature.beacon) > 0:\n            gap = feature.beacon.loc[0, 'timestamp2'] - \\\n                feature.beacon.loc[0, 'timestamp']\n        else:\n            gap = (feature.wifi.last_seen_timestamp.values -\n                   feature.wifi.timestamp.values).max()+210.14426803816337  # from mean gap\n        site_id = feature.site_id\n        train_meta_site = train_meta[train_meta.site_id == site_id]\n        \n        #postprocess start point based on leakage\n        train_meta_site_end = train_meta_site[(\n            start_time+gap) > train_meta_site.end_time]\n        if len(train_meta_site_end) > 0:\n            nearest_endpoint = train_meta_site_end.loc[train_meta_site_end.end_time.idxmax(\n            )]\n            if postprocess_start and (start_time + gap - nearest_endpoint.end_time < start_threshold):\n                out_df.loc[(out_df.path_id == path_id) & (out_df.timestamp == \n                    out_df[out_df.path_id == path_id].timestamp.min()), 'x'] = nearest_endpoint.end_wp_x\n                out_df.loc[(out_df.path_id == path_id) & (out_df.timestamp == \n                    out_df[out_df.path_id == path_id].timestamp.min()), 'y'] = nearest_endpoint.end_wp_y\n                start_counter += 1\n        \n        #postprocess end point based on leakage\n        train_meta_site_start = train_meta_site[train_meta_site.start_time > (\n            end_time+gap)]\n        if len(train_meta_site_start) > 0:\n            nearest_startpoint = train_meta_site_start.loc[train_meta_site_start.start_time.idxmin(\n            )]\n            if postprocess_end and (nearest_startpoint.start_time - end_time - gap < end_threshold):\n                out_df.loc[(out_df.path_id == path_id) & (out_df.timestamp == \n                    out_df[out_df.path_id == path_id].timestamp.max()), 'x'] = nearest_startpoint.start_wp_x\n                out_df.loc[(out_df.path_id == path_id) & (out_df.timestamp == \n                    out_df[out_df.path_id == path_id].timestamp.max()), 'y'] = nearest_startpoint.start_wp_y\n                end_counter += 1\n                \n        #postprocess floor based on leakage\n        if postprocess_floor:\n            if (len(train_meta_site_end) > 0) and (len(train_meta_site_start) > 0) and (nearest_endpoint.n_floor == nearest_startpoint.n_floor):\n                out_df.loc[(out_df.path_id == path_id),\n                                  'floor'] = nearest_endpoint.n_floor\n                floor_counter += (out_df.path_id == path_id).sum()\n\n            # uncomment this section if you want to postprocess all floor predictions\n            # elif (len(train_meta_site_end) > 0) and (len(train_meta_site_start) > 0):\n            #     diff_start_time = start_time - nearest_endpoint.end_time\n            #     diff_end_time = nearest_startpoint.start_time - end_time\n            #     if diff_start_time < diff_end_time:\n            #         out_df.loc[(out_df.path_id == path_id),\n            #                           'floor'] = nearest_endpoint.n_floor\n            #         floor_counter += (out_df.path_id == path_id).sum()\n            #     if diff_end_time < diff_start_time:\n            #         out_df.loc[(out_df.path_id == path_id),\n            #                           'floor'] = nearest_startpoint.n_floor\n            #         floor_counter += (out_df.path_id == path_id).sum()\n            # elif len(train_meta_site_end) > 0:\n            #     out_df.loc[(out_df.path_id == path_id),\n            #                       'floor'] = nearest_endpoint.n_floor\n            #     floor_counter += (out_df.path_id == path_id).sum()\n            # elif len(train_meta_site_start) > 0:\n            #     out_df.loc[(out_df.path_id == path_id),\n            #                       'floor'] = nearest_startpoint.n_floor\n            #     floor_counter += (out_df.path_id == path_id).sum()\n            \n    print(str(start_counter) + ' start points are postprocessed.')\n    print(str(end_counter) + ' end points are postprocessed.')\n    print(str(floor_counter) + ' floors are postprocessed.')\n    out_df = out_df.drop(\n        [\"site_id\", \"path_id\", \"timestamp\"], axis=1)\n    return out_df","metadata":{"_uuid":"88af2d0b-394b-44eb-929c-3c3e75ec40ff","_cell_guid":"8451018f-6542-4583-a862-539609e29546","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission_df = pd.read_csv('../input/3-3-g6-indoor-navigation-snap-to-grid/submission_snap_to_grid.csv')\nsubmission_df = pd.read_csv('first-submission.csv')","metadata":{"_uuid":"e4bf026f-5f20-47f3-9789-2665e2b2dd0b","_cell_guid":"6578ce26-222c-442c-aa03-439a8836956e","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I used the submission file of the current best public notebook for demonstration.","metadata":{"_uuid":"037c7540-b1d1-40b9-9ec2-5779d85ffd06","_cell_guid":"a3cb94c5-b69a-47c3-b202-e3e641443919","trusted":true}},{"cell_type":"code","source":"submission_df_leak_start = leak_postprocessing(submission_df,train_meta_sub, postprocess_start=True, postprocess_end=False, postprocess_floor=False)\nsubmission_df_leak_start.to_csv(\n    'submission_df_leak_start.csv', index=False)","metadata":{"_uuid":"3e4f48f9-7bc5-4976-b714-40bb61a7c51a","_cell_guid":"38d91f97-bcbc-457c-b186-4ff6e58b4f17","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I postprocessed start waypoints only here.","metadata":{"_uuid":"d5741102-4bf4-470c-b324-978d513d5764","_cell_guid":"1c6daf14-58ef-4bbb-afc1-81bcfddd1fc0","trusted":true}},{"cell_type":"code","source":"submission_df_leak_end = leak_postprocessing(submission_df,train_meta_sub, postprocess_start=False, postprocess_end=True, postprocess_floor=False)\nsubmission_df_leak_end.to_csv(\n    'submission_df_leak_end.csv', index=False)","metadata":{"_uuid":"4f301577-5154-4c04-b278-1f4519e14fa4","_cell_guid":"12e0712f-2135-4e28-8338-5f6df3591687","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I postprocessed end waypoints only here.","metadata":{"_uuid":"a4624cf3-e2a5-4bec-9332-c9685f388301","_cell_guid":"cd1bce5a-d667-4b79-ba46-3a973a95a22b","trusted":true}},{"cell_type":"code","source":"submission_df_leak_floor = leak_postprocessing(submission_df,train_meta_sub, postprocess_start=False, postprocess_end=False, postprocess_floor=True)\nsubmission_df_leak_floor.to_csv(\n    'submission_df_leak_floor.csv', index=False)","metadata":{"_uuid":"c8676725-9b4a-4f19-95a9-5b5c06b11a7a","_cell_guid":"49451039-3045-49a2-a155-6a52abf6c729","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I postprocessed floor only here","metadata":{"_uuid":"27daf023-3a7f-48ae-bd1e-62f1c225fd84","_cell_guid":"7b1db20b-c4fd-4030-9c37-098ec8792df4","trusted":true}},{"cell_type":"code","source":"submission_df_leak_all = leak_postprocessing(submission_df,train_meta_sub, postprocess_start=True, postprocess_end=True, postprocess_floor=True)\nsubmission_df_leak_all.to_csv(\n    'submission_df_leak_all.csv', index=False)","metadata":{"_uuid":"ee26ec35-13f9-49c7-81a4-fb3a3b7b2c9b","_cell_guid":"ee8a91b6-325d-4a3a-b95f-0203cd031a48","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I postprocessed all data here.","metadata":{"_uuid":"a420fbc9-1da5-42d7-9feb-c1247a80fcac","_cell_guid":"1215e609-808c-4765-8760-ae2e3785ba76","trusted":true}},{"cell_type":"code","source":"submission_df_leak_start_end = leak_postprocessing(submission_df,train_meta_sub, postprocess_start=True, postprocess_end=True, postprocess_floor=False)\nsubmission_df_leak_start_end.to_csv(\n    'submission_df_leak_start_end.csv', index=False)","metadata":{"_uuid":"ecdbf324-1fd1-46fc-a9b1-c69229603416","_cell_guid":"0f795ac2-9039-4de1-9269-3d5d27eea311","collapsed":false,"jupyter":{"outputs_hidden":false},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I postprocessed start and end point here.","metadata":{"_uuid":"1afe3a54-9448-4598-8ee4-a22b4dcfe6aa","_cell_guid":"0bbdeb05-865d-4d13-9a5d-5c14e649dc11","trusted":true}},{"cell_type":"markdown","source":"The LB Scores are as follows\n* Original: 4.789\n* Start point: 4.751\n* End point: 4.750\n* Floor: 4.995\n* All: 4.924\n* Start and end point: 4.718\n\nAlthough the postprocessing floor makes the LB score a bit lower, it is surprising that the score is comparable even if about 80% of floor predictions were replaced with those without any sensor information. If I uncomment the section in postprocessing function and postprocess all floor predictions, the LB score becomes 8.7xx. It is still not so bad, if I consider the fact that LB score < 15 means my floor prediction error < 1.\n\nAny comments are welcome.","metadata":{"_uuid":"399d013f-e64d-4a0c-a3d0-04e21582cb88","_cell_guid":"3d16a0d3-3bbd-4594-9d34-8828dd3e9782","trusted":true}}]}