{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this notebook I do EDA on devices. After that, I show you how to improve the accuracy by removing location information from devices with poor accuracy and interpolate it from other devices.\n\nこのノートブックではdeviceに関するEDAを行い、精度が高くないdeviceの位置情報を取り除き他のdeviceからの位置情報を補間することで精度を向上させる方法を紹介します。","metadata":{}},{"cell_type":"markdown","source":"Score(Public Leaderboard)  \n[Baseline post-processing by outlier correction](https://www.kaggle.com/dehokanta/baseline-post-processing-by-outlier-correction) by [dehokanta](https://www.kaggle.com/dehokanta) : 6.164  \nthis notebook: 6.089\n\n**0.075 up!**","metadata":{}},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"markdown","source":"Let's look at the accuracy for each device. You can see that the accuracy varies from device to device.\n\ndeviceごとの精度を見てみましょう。device間で精度が異なるのがわかります。","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom pathlib import Path\nfrom glob import glob\n\n\ndef get_groundtruth(path: Path) -> pd.DataFrame:\n    output_df = pd.DataFrame()\n    \n    for path in glob(str(path / 'train/*/*/ground_truth.csv')):\n        _df = pd.read_csv(path)\n        output_df = pd.concat([output_df, _df])\n    output_df = output_df.reset_index(drop=True)\n    \n    _columns = ['latDeg', 'lngDeg', 'heightAboveWgs84EllipsoidM']\n    output_df[['t_'+col for col in _columns]] = output_df[_columns]\n    output_df = output_df.drop(columns=_columns, axis=1)\n    return output_df\n\n\ndef calc_haversine(lat1, lon1, lat2, lon2):\n    \"\"\"Calculates the great circle distance between two points\n    on the earth. Inputs are array-like and specified in decimal degrees.\n    \"\"\"\n    lat1, lon1, lat2, lon2 = map(np.radians, [lat1, lon1, lat2, lon2])\n    dlat = lat2 - lat1\n    dlon = lon2 - lon1\n    a = np.sin(dlat/2.0)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2.0)**2\n\n    c = 2 * np.arcsin(a**0.5)\n    dist = 6_367_000 * c\n    return dist\n\n\ndef check_score(input_df: pd.DataFrame) -> pd.DataFrame:\n    output_df = input_df.copy()\n    \n    output_df['meter'] = input_df.apply(\n        lambda r: calc_haversine(\n            r.latDeg, r.lngDeg, r.t_latDeg, r.t_lngDeg\n        ),\n        axis=1\n    )\n\n    meter_score = output_df['meter'].mean()\n    print(f'error meter: {meter_score}')\n\n    scores = []\n    for phone in output_df['phone'].unique():\n        _index = output_df['phone']==phone\n        p_50 = np.percentile(output_df.loc[_index, 'meter'], 50)\n        p_95 = np.percentile(output_df.loc[_index, 'meter'], 95)\n        scores.append(p_50)\n        scores.append(p_95)\n\n    score = sum(scores) / len(scores)\n    print(f'score: {score}')\n    \n    return output_df","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:26:49.700427Z","iopub.execute_input":"2021-06-07T05:26:49.700851Z","iopub.status.idle":"2021-06-07T05:26:49.715972Z","shell.execute_reply.started":"2021-06-07T05:26:49.700817Z","shell.execute_reply":"2021-06-07T05:26:49.714913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# read data\nBASE_DIR = Path('../input/google-smartphone-decimeter-challenge')\ntrain_base = pd.read_csv(BASE_DIR / 'baseline_locations_train.csv')\ntest_base = pd.read_csv(BASE_DIR / 'baseline_locations_test.csv')\n\n# merge graoundtruth\ntrain_base = train_base.merge(\n    get_groundtruth(BASE_DIR),\n    on=['collectionName', 'phoneName', 'millisSinceGpsEpoch']\n)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:31.305861Z","iopub.execute_input":"2021-06-07T05:24:31.306287Z","iopub.status.idle":"2021-06-07T05:24:32.621912Z","shell.execute_reply.started":"2021-06-07T05:24:31.306242Z","shell.execute_reply":"2021-06-07T05:24:32.620866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check score\ntrain_base = check_score(train_base)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:32.623387Z","iopub.execute_input":"2021-06-07T05:24:32.62383Z","iopub.status.idle":"2021-06-07T05:24:43.287616Z","shell.execute_reply.started":"2021-06-07T05:24:32.623786Z","shell.execute_reply":"2021-06-07T05:24:43.286481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show boxplot\nfor name, df in train_base.groupby('collectionName'):    \n    sns.boxplot(data=df, x='phoneName', y='meter', width=0.5)\n    plt.title(name)\n    plt.ylim(0, 20)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:43.288891Z","iopub.execute_input":"2021-06-07T05:24:43.289137Z","iopub.status.idle":"2021-06-07T05:24:48.115573Z","shell.execute_reply.started":"2021-06-07T05:24:43.289113Z","shell.execute_reply":"2021-06-07T05:24:48.114415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show describe\ntrain_base.groupby('phoneName')['meter'].describe().T","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:48.117165Z","iopub.execute_input":"2021-06-07T05:24:48.117636Z","iopub.status.idle":"2021-06-07T05:24:48.170543Z","shell.execute_reply.started":"2021-06-07T05:24:48.117564Z","shell.execute_reply":"2021-06-07T05:24:48.169556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you look at the data, you can see that the error is different for devices with the same collectionName.  \nNow, let's interpolate the location information from other devices except for SamsungS20Ultra which has a lot of errors.\n\nデータを見てみると、同じcollectionNameでもdeviceによって誤差が異なることがわかります。  \n今回は誤差を多く含むSamsungS20Ultraの除いてほかのdeviceの位置情報から補間してみましょう。","metadata":{}},{"cell_type":"markdown","source":"# Method","metadata":{}},{"cell_type":"code","source":"def get_removedevice(input_df: pd.DataFrame, divece: str) -> pd.DataFrame:\n    input_df['index'] = input_df.index\n    input_df = input_df.sort_values('millisSinceGpsEpoch')\n    input_df.index = input_df['millisSinceGpsEpoch'].values\n\n    output_df = pd.DataFrame() \n    for _, subdf in input_df.groupby('collectionName'):\n\n        phones = subdf['phoneName'].unique()\n\n        if (len(phones) == 1) or (not divece in phones):\n            output_df = pd.concat([output_df, subdf])\n            continue\n\n        origin_df = subdf.copy()\n        \n        _index = subdf['phoneName']==divece\n        subdf.loc[_index, 'latDeg'] = np.nan\n        subdf.loc[_index, 'lngDeg'] = np.nan\n        subdf = subdf.interpolate(method='index', limit_area='inside')\n\n        _index = subdf['latDeg'].isnull()\n        subdf.loc[_index, 'latDeg'] = origin_df.loc[_index, 'latDeg'].values\n        subdf.loc[_index, 'lngDeg'] = origin_df.loc[_index, 'lngDeg'].values\n\n        output_df = pd.concat([output_df, subdf])\n\n    output_df.index = output_df['index'].values\n    output_df = output_df.sort_index()\n\n    del output_df['index']\n    \n    return output_df","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:48.173753Z","iopub.execute_input":"2021-06-07T05:24:48.174133Z","iopub.status.idle":"2021-06-07T05:24:48.184473Z","shell.execute_reply.started":"2021-06-07T05:24:48.174101Z","shell.execute_reply":"2021-06-07T05:24:48.183259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_remove = get_removedevice(train_base, 'SamsungS20Ultra')","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:48.186282Z","iopub.execute_input":"2021-06-07T05:24:48.186784Z","iopub.status.idle":"2021-06-07T05:24:48.779818Z","shell.execute_reply.started":"2021-06-07T05:24:48.18674Z","shell.execute_reply":"2021-06-07T05:24:48.778681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"\ncheck_score(train_base):\n    error meter: 3.846848374995186\n    score: 5.287970649047862\n\"\"\"\n\ntrain_remove = check_score(train_remove)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:24:48.780992Z","iopub.execute_input":"2021-06-07T05:24:48.781295Z","iopub.status.idle":"2021-06-07T05:24:59.533639Z","shell.execute_reply.started":"2021-06-07T05:24:48.781268Z","shell.execute_reply":"2021-06-07T05:24:59.532764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As described above, I was able to increase the accuracy easily.  \nThis is a post processing step, so you can improve the accuracy even without train data. To try it out, let's use [Baseline post-processing by outlier correction](https://www.kaggle.com/dehokanta/baseline-post-processing-by-outlier-correction) created by [dehokanta](https://www.kaggle.com/dehokanta).\n\n以上のように簡単に精度を向上することができました。  \nこれは後処理なのでtrainデータがなくても精度が向上させることができます。試しに[dehokanta](https://www.kaggle.com/dehokanta)さんの[Baseline post-processing by outlier correction](https://www.kaggle.com/dehokanta/baseline-post-processing-by-outlier-correction)に対して使ってみましょう。","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('../input/baseline-post-processing-by-outlier-correction/submission.csv')\nsubmission['collectionName'] = submission['phone'].map(lambda x: x.split('_')[0])\nsubmission['phoneName'] = submission['phone'].map(lambda x: x.split('_')[1])\nsubmission = get_removedevice(submission, 'SamsungS20Ultra')\n\nsubmission = submission.drop(columns=['collectionName', 'phoneName'], axis=1)\nsubmission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-07T05:56:35.075328Z","iopub.execute_input":"2021-06-07T05:56:35.075709Z","iopub.status.idle":"2021-06-07T05:56:36.182686Z","shell.execute_reply.started":"2021-06-07T05:56:35.075678Z","shell.execute_reply":"2021-06-07T05:56:36.181635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can only process the submission file, but be sure to process the train data as well to check the score.  \nThank you for reading to the end. I look forward to your questions and comments.\n\nサブミッションファイルに対してのみ処理することも可能ですが、確実にtrainデータに対しても処理を行いスコアを確認してください。  \n最後まで読んでいただきありがとうございました！質問とコメントお待ちしています。","metadata":{}},{"cell_type":"markdown","source":"# Reference\n\nI used these notebooks as a reference in creating this. Thank you very much!\n\n* [Baseline post-processing by outlier correction](https://www.kaggle.com/dehokanta/baseline-post-processing-by-outlier-correction) created by [dehokanta](https://www.kaggle.com/dehokanta).","metadata":{}}]}