{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## **Overview**\n\nOne of the important things in this competition is how we can **reduce the error of multipath.**\n\n>**[Multipath](https://gssc.esa.int/navipedia/index.php/Multipath)**\n>\n>The interference by multipath is generated when a signal arrives, by different ways, at the antenna (see figure 1). Its principal cause is the antenna closeness to the reflecting structures, and it is important when the signal comes from the satellite with low elevation.\n\nAccording to [this article](https://www.kaggle.com/t88take/gsdc-eda-error-when-stopping), the phenomenon of multipath occurs when the speed is zero.\n\nSo I created a model to determine if a car is moving using IMU data and was able to record an accuracy of 94%, and I'm going to publish it in code.\n\nI hope this will be useful for everyone who is participating in the competition！！！\n\n![image](https://user-images.githubusercontent.com/47235292/127402718-0e8b4075-322a-41a4-86a7-e321e86bd2e5.png)","metadata":{}},{"cell_type":"markdown","source":"**Reference¶**\n\n@T88's notebook https://www.kaggle.com/t88take/gsdc-eda-error-when-stopping\n\n@museas's notebook https://www.kaggle.com/museas/estimating-the-direction-with-a-magnetic-sensor\n\nData Overview https://www.kaggle.com/c/google-smartphone-decimeter-challenge/data","metadata":{}},{"cell_type":"markdown","source":"## Light GBM with IMU data\n\n----------------------------\nPlease note that the code is absolutly messy m(__)m\n\n----------------------------\n\nAs **training data**, two of the downtown paths of the train data were used.\n\nAs **test data**, one of the remaining downtown paths of the train data was used \n\nFor the input features, I used IMU acceleration, gyro and magn.No pre-processing was applied to each feature. This may have come as a surprise to you.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport pathlib\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:51:33.564782Z","iopub.execute_input":"2021-07-28T16:51:33.565423Z","iopub.status.idle":"2021-07-28T16:51:33.576845Z","shell.execute_reply.started":"2021-07-28T16:51:33.5653Z","shell.execute_reply":"2021-07-28T16:51:33.575411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Method:Make train data","metadata":{}},{"cell_type":"code","source":"#  making ground truth file\ndef make_gt(path, collectionName, phoneName):\n    # ground_truth\n    p = pathlib.Path(path)\n    gt_files = list(p.glob('train/*/*/ground_truth.csv'))\n\n    gts = []\n    for gt_file in gt_files:\n        gts.append(pd.read_csv(gt_file))\n    ground_truth = pd.concat(gts)\n    \n    # baseline\n    cols = ['collectionName', 'phoneName', 'millisSinceGpsEpoch', 'latDeg', 'lngDeg']\n    baseline = pd.read_csv(path + '/baseline_locations_train.csv', usecols=cols)\n    ground_truth = ground_truth.merge(baseline, how='inner', on=cols[:3], suffixes=('_gt', '_bs'))\n    ground_truth[\"millisSinceGpsEpoch\"] = ground_truth[\"millisSinceGpsEpoch\"]//1000\n    if (collectionName is None) or (phoneName is None):\n        return ground_truth\n    else:\n        return ground_truth[(ground_truth['collectionName'] == collectionName) & (ground_truth['phoneName'] == phoneName)]\n    \n\ndef make_tag(df, tag_v):\n    df.loc[df['speedMps'] < tag_v, 'tag'] = 1\n    df.loc[df['speedMps'] >= tag_v, 'tag'] = 0\n    return df\n\n\n# loading gnss file\ndef gnss_log_to_dataframes(path):\n    print('Loading ' + path, flush=True)\n    gnss_section_names = {'Raw', 'UncalAccel', 'UncalGyro', 'UncalMag', 'Fix', 'Status', 'OrientationDeg'}\n    with open(path) as f_open:\n        datalines = f_open.readlines()\n\n    datas = {k: [] for k in gnss_section_names}\n    gnss_map = {k: [] for k in gnss_section_names}\n    for dataline in datalines:\n        is_header = dataline.startswith('#')\n        dataline = dataline.strip('#').strip().split(',')\n        # skip over notes, version numbers, etc\n        if is_header and dataline[0] in gnss_section_names:\n            try:\n                gnss_map[dataline[0]] = dataline[1:]\n            except:\n                pass\n        elif not is_header:\n            try:\n                datas[dataline[0]].append(dataline[1:])\n            except:\n                pass\n    results = dict()\n    for k, v in datas.items():\n        results[k] = pd.DataFrame(v, columns=gnss_map[k])\n    # pandas doesn't properly infer types from these lists by default\n    for k, df in results.items():\n        for col in df.columns:\n            if col == 'CodeType':\n                continue\n            try:\n                results[k][col] = pd.to_numeric(results[k][col])\n            except:\n                pass\n    return results\n\n\ndef add_IMU(df, INPUT, cname, pname):\n    path = INPUT + \"/train/\"+cname+\"/\"+pname+\"/\"+pname+\"_GnssLog.txt\"\n    gnss_dfs = gnss_log_to_dataframes(path)\n    acce_df = gnss_dfs[\"UncalAccel\"]\n    magn_df = gnss_dfs[\"UncalMag\"]\n    gyro_df = gnss_dfs[\"UncalGyro\"]\n    \n    acce_df[\"millisSinceGpsEpoch\"] = acce_df[\"utcTimeMillis\"] - 315964800000\n    acce_df[\"millisSinceGpsEpoch\"] = acce_df[\"millisSinceGpsEpoch\"]//1000 +18\n    magn_df[\"millisSinceGpsEpoch\"] = magn_df[\"utcTimeMillis\"] - 315964800000\n    magn_df[\"millisSinceGpsEpoch\"] = magn_df[\"millisSinceGpsEpoch\"]//1000 +18\n    gyro_df[\"millisSinceGpsEpoch\"] = gyro_df[\"utcTimeMillis\"] - 315964800000\n    gyro_df[\"millisSinceGpsEpoch\"] = gyro_df[\"millisSinceGpsEpoch\"]//1000 +18\n    \n    acce_df[\"x_f_acce\"] = acce_df[\"UncalAccelZMps2\"]\n    acce_df[\"y_f_acce\"] = acce_df[\"UncalAccelXMps2\"]\n    acce_df[\"z_f_acce\"] = acce_df[\"UncalAccelYMps2\"]\n    # magn \n    magn_df[\"x_f_magn\"] = magn_df[\"UncalMagZMicroT\"]\n    magn_df[\"y_f_magn\"] = magn_df[\"UncalMagYMicroT\"]\n    magn_df[\"z_f_magn\"] = magn_df[\"UncalMagXMicroT\"]\n    # gyro\n    gyro_df[\"x_f_gyro\"] = gyro_df[\"UncalGyroXRadPerSec\"]\n    gyro_df[\"y_f_gyro\"] = gyro_df[\"UncalGyroYRadPerSec\"]\n    gyro_df[\"z_f_gyro\"] = gyro_df[\"UncalGyroZRadPerSec\"]    \n\n    df = pd.merge_asof(df[[\"collectionName\", \"phoneName\", \"millisSinceGpsEpoch\", \"latDeg_gt\", \"lngDeg_gt\", \"latDeg_bs\", \"lngDeg_bs\", \"heightAboveWgs84EllipsoidM\", \"speedMps\"]].sort_values('millisSinceGpsEpoch'), acce_df[[\"millisSinceGpsEpoch\", \"x_f_acce\", \"y_f_acce\", \"z_f_acce\"]].sort_values('millisSinceGpsEpoch'), on='millisSinceGpsEpoch', direction='nearest')\n    df = pd.merge_asof(df[[\"collectionName\", \"phoneName\", \"millisSinceGpsEpoch\", \"latDeg_gt\", \"lngDeg_gt\", \"latDeg_bs\", \"lngDeg_bs\", \"heightAboveWgs84EllipsoidM\", \"speedMps\", \"x_f_acce\", \"y_f_acce\", \"z_f_acce\"]].sort_values('millisSinceGpsEpoch'), magn_df[[\"millisSinceGpsEpoch\", \"x_f_magn\", \"y_f_magn\", \"z_f_magn\"]].sort_values('millisSinceGpsEpoch'), on='millisSinceGpsEpoch', direction='nearest')\n    df = pd.merge_asof(df[[\"collectionName\", \"phoneName\", \"millisSinceGpsEpoch\", \"latDeg_gt\", \"lngDeg_gt\", \"latDeg_bs\", \"lngDeg_bs\", \"heightAboveWgs84EllipsoidM\", \"speedMps\", \"x_f_acce\", \"y_f_acce\", \"z_f_acce\", \"x_f_magn\", \"y_f_magn\", \"z_f_magn\"]].sort_values('millisSinceGpsEpoch'), gyro_df[[\"millisSinceGpsEpoch\", \"x_f_gyro\", \"y_f_gyro\", \"z_f_gyro\"]].sort_values('millisSinceGpsEpoch'), on='millisSinceGpsEpoch', direction='nearest')\n    return df\n\ndef make_train(INPUT, train_cname, tag_v):\n    # make ground_truth file\n    gt = make_gt(INPUT, None, None)\n    train_df = pd.DataFrame()\n    for cname in train_cname:\n        phone_list = gt[gt['collectionName'] == cname]['phoneName'].drop_duplicates()\n        for pname in phone_list:\n            df = gt[(gt['collectionName'] == cname) & (gt['phoneName'] == pname)]\n            df = add_IMU(df, INPUT, cname, pname)\n            train_df = pd.concat([train_df, df])\n    # make tag\n    train_df = make_tag(train_df, tag_v)\n    return train_df","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:52:12.860517Z","iopub.execute_input":"2021-07-28T16:52:12.861043Z","iopub.status.idle":"2021-07-28T16:52:12.889071Z","shell.execute_reply.started":"2021-07-28T16:52:12.861009Z","shell.execute_reply":"2021-07-28T16:52:12.888305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"###  Method:Model(Light GBM)","metadata":{}},{"cell_type":"code","source":"import lightgbm as lgb\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import accuracy_score\n\n\ndef lgbm(train, test, col, lgb_params):\n    model = lgb.LGBMClassifier(**lgb_params)\n    model.fit(train[col], train['tag'])\n    preds = model.predict(test[col])\n    print('confusion matrix :  \\n', confusion_matrix(preds, test['tag']))\n    print('accuracy score : ', accuracy_score(preds, test['tag']))\n    return preds","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:52:13.3021Z","iopub.execute_input":"2021-07-28T16:52:13.302609Z","iopub.status.idle":"2021-07-28T16:52:13.308476Z","shell.execute_reply.started":"2021-07-28T16:52:13.302578Z","shell.execute_reply":"2021-07-28T16:52:13.307758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Method:Confirm Score","metadata":{}},{"cell_type":"code","source":"def get_train_score(df):\n    # calc_distance_error\n    df['err'] =  calc_haversine(df.latDeg_bs, df.lngDeg_bs, \n    df.latDeg_gt, df.lngDeg_gt)\n    # calc_evaluate_score\n    df['phone'] = df['collectionName'] + '_' + df['phoneName']\n    res = df.groupby('phone')['err'].agg([percentile50, percentile95])\n    res['p50_p90_mean'] = (res['percentile50'] + res['percentile95']) / 2 \n    score = res['p50_p90_mean'].mean()\n    return score\n\n\ndef percentile50(x):\n    return np.percentile(x, 50)\n\n\ndef percentile95(x):\n    return np.percentile(x, 95)\n\n\ndef calc_haversine(lat1, lon1, lat2, lon2):\n    \"\"\"Calculates the great circle distance between two points\n    on the earth. Inputs are array-like and specified in decimal degrees.\n    \"\"\"\n    lat1, lon1, lat2, lon2 = map(np.radians, [lat1, lon1, lat2, lon2])\n    dlat = lat2 - lat1\n    dlon = lon2 - lon1\n    a = np.sin(dlat/2.0)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2.0)**2\n\n    c = 2 * np.arcsin(a**0.5)\n    dist = 6_367_000 * c\n    return dist","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:52:13.628592Z","iopub.execute_input":"2021-07-28T16:52:13.629349Z","iopub.status.idle":"2021-07-28T16:52:13.638985Z","shell.execute_reply.started":"2021-07-28T16:52:13.629288Z","shell.execute_reply":"2021-07-28T16:52:13.63822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Initial value","metadata":{}},{"cell_type":"markdown","source":"Please adjust the parameters as you like.","metadata":{}},{"cell_type":"code","source":"INPUT = '../input/google-smartphone-decimeter-challenge'\n\ntrain_cname = ['2021-04-29-US-SJC-2', '2021-03-10-US-SVL-1']\ntest_cname = ['2021-04-28-US-SJC-1']\ntag_v = 0.5\ncol = [\"x_f_acce\", \"y_f_acce\", \"z_f_acce\", \"x_f_magn\", \"y_f_magn\", \"z_f_magn\", \"x_f_gyro\", \"y_f_gyro\", \"z_f_gyro\"]\n\n# parameter\nlgb_params = {\n    'num_leaves': 90,\n    'n_estimators': 125,\n}","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:52:14.374448Z","iopub.execute_input":"2021-07-28T16:52:14.375114Z","iopub.status.idle":"2021-07-28T16:52:14.380961Z","shell.execute_reply.started":"2021-07-28T16:52:14.375076Z","shell.execute_reply":"2021-07-28T16:52:14.379718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Main","metadata":{}},{"cell_type":"code","source":"# make train&test\ntrain_df = make_train(INPUT, train_cname, tag_v)\ntest_df = make_train(INPUT, test_cname, tag_v)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:52:14.858392Z","iopub.execute_input":"2021-07-28T16:52:14.858828Z","iopub.status.idle":"2021-07-28T16:54:21.936779Z","shell.execute_reply.started":"2021-07-28T16:52:14.858793Z","shell.execute_reply":"2021-07-28T16:54:21.935543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prediction with light gbm\ntest_df['preds'] = lgbm(train_df, test_df, col, lgb_params)","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:54:21.938691Z","iopub.execute_input":"2021-07-28T16:54:21.93903Z","iopub.status.idle":"2021-07-28T16:54:22.592745Z","shell.execute_reply.started":"2021-07-28T16:54:21.938997Z","shell.execute_reply":"2021-07-28T16:54:22.591716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize data\n\nVisualizing that speed is 0 or not.\n\nLooking at the figure, it appears that multipath is occurring where the car is determined to be stationary😄","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\nfig = px.scatter_mapbox(test_df,\n                    # Here, plotly gets, (x,y) coordinates\n                    lat=\"latDeg_bs\",\n                    lon=\"lngDeg_bs\",\n                    text='phoneName',\n\n                    #Here, plotly detects color of series\n                    color=\"preds\",\n                    labels=\"collectionName\",\n\n                    zoom=14.5,\n                    center={\"lat\":37.334, \"lon\":-121.89},\n                    height=600,\n                    width=800)\nfig.update_layout(mapbox_style='stamen-terrain')\nfig.update_layout(margin={\"r\": 0, \"t\": 0, \"l\": 0, \"b\": 0})\nfig.update_layout(title_text=\"GPS trafic\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-28T16:54:27.997015Z","iopub.execute_input":"2021-07-28T16:54:27.997699Z","iopub.status.idle":"2021-07-28T16:54:28.12948Z","shell.execute_reply.started":"2021-07-28T16:54:27.997645Z","shell.execute_reply":"2021-07-28T16:54:28.128296Z"},"trusted":true},"execution_count":null,"outputs":[]}]}