{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview\n\nGeocoding is the process of taking a text-based description of a location, and returning geographic coordinates, frequently latitude/longitude pairs. Reverse geocoding is the process of converting a location as described by geographic coordinates (latitude, longitude) to a human-readable address or place. \n\nHow can this be useful? Well, we are given latitude-longitude coordinates, so we can perform reverse geocoding. As a result, we can generate lots of new data that could potentially be used as model features or inputs to post processing.\n\nIs this Competition Viable? To put it shortly, yes. As stated by Rule 7c, external data is allowed as long as it is publicly available. Since this data is generated using an open source library, it is allowed.\n\nI hope you find this useful and if you do, leave an upvote.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport json\nimport os\nimport glob\n\nimport matplotlib.pyplot as plt\nfrom matplotlib_venn import venn2, venn2_circles\nimport seaborn as sns\nfrom tqdm.notebook import tqdm\nimport pathlib\nimport plotly\nimport plotly.express as px\nfrom pathlib import Path\nimport pyproj\nfrom pyproj import Proj, transform\nfrom geopy.geocoders import Nominatim\nimport plotly.express as px\nfrom scipy import stats\n\n!pip install reverse_geocoder\nimport reverse_geocoder as rg","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-26T20:25:58.699360Z","iopub.execute_input":"2021-06-26T20:25:58.699688Z","iopub.status.idle":"2021-06-26T20:26:12.184665Z","shell.execute_reply.started":"2021-06-26T20:25:58.699660Z","shell.execute_reply":"2021-06-26T20:26:12.183591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INPUT = '../input/google-smartphone-decimeter-challenge'\nbase_train = pd.read_csv(INPUT + '/' + 'baseline_locations_train.csv')\nbase_test = pd.read_csv(INPUT + '/' + 'baseline_locations_test.csv')\nsample_sub = pd.read_csv(INPUT + '/' + 'sample_submission.csv')\nbase_test.head(1)","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:12.186514Z","iopub.execute_input":"2021-06-26T20:26:12.186816Z","iopub.status.idle":"2021-06-26T20:26:12.811792Z","shell.execute_reply.started":"2021-06-26T20:26:12.186786Z","shell.execute_reply":"2021-06-26T20:26:12.810817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ground_truth\np = pathlib.Path(INPUT)\ngt_files = list(p.glob('train/*/*/ground_truth.csv'))\nprint('ground_truth.csv count : ', len(gt_files))\n\ngts = []\nfor gt_file in tqdm(gt_files):\n    gts.append(pd.read_csv(gt_file))\nground_truth = pd.concat(gts)\n\n#display(ground_truth.head())","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:12.813717Z","iopub.execute_input":"2021-06-26T20:26:12.814010Z","iopub.status.idle":"2021-06-26T20:26:13.909474Z","shell.execute_reply.started":"2021-06-26T20:26:12.813982Z","shell.execute_reply":"2021-06-26T20:26:13.908678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reverse Goecoder Setup","metadata":{}},{"cell_type":"code","source":"base_test[\"geom\"] = base_test[\"latDeg\"].map(str) + \",\" + base_test[\"lngDeg\"].map(str)","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:13.910886Z","iopub.execute_input":"2021-06-26T20:26:13.911188Z","iopub.status.idle":"2021-06-26T20:26:14.117253Z","shell.execute_reply.started":"2021-06-26T20:26:13.911160Z","shell.execute_reply":"2021-06-26T20:26:14.116544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def geocoder(data):\n    locator = Nominatim(user_agent=\"myGeocoder\")\n    coordinates = data['geom']\n    location = locator.reverse(coordinates)\n    return location.raw","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:14.118665Z","iopub.execute_input":"2021-06-26T20:26:14.119320Z","iopub.status.idle":"2021-06-26T20:26:14.124357Z","shell.execute_reply.started":"2021-06-26T20:26:14.119279Z","shell.execute_reply":"2021-06-26T20:26:14.123456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA with Reverse Geocoder","metadata":{}},{"cell_type":"markdown","source":"### Example 1","metadata":{}},{"cell_type":"code","source":"# I am using the first point to demonstrate\nex_point = base_test.iloc[0]\nex_point","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:14.125648Z","iopub.execute_input":"2021-06-26T20:26:14.125943Z","iopub.status.idle":"2021-06-26T20:26:14.139978Z","shell.execute_reply.started":"2021-06-26T20:26:14.125919Z","shell.execute_reply":"2021-06-26T20:26:14.139116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_output = geocoder(ex_point)\nex_output","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:14.141855Z","iopub.execute_input":"2021-06-26T20:26:14.142282Z","iopub.status.idle":"2021-06-26T20:26:14.520408Z","shell.execute_reply.started":"2021-06-26T20:26:14.142242Z","shell.execute_reply":"2021-06-26T20:26:14.519516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Example 2","metadata":{}},{"cell_type":"code","source":"# I am using a random point to demonstrate\nex_point = base_test.iloc[420]\nex_point","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:14.521499Z","iopub.execute_input":"2021-06-26T20:26:14.521765Z","iopub.status.idle":"2021-06-26T20:26:14.528675Z","shell.execute_reply.started":"2021-06-26T20:26:14.521741Z","shell.execute_reply":"2021-06-26T20:26:14.527681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_output = geocoder(ex_point)\nex_output","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:14.531332Z","iopub.execute_input":"2021-06-26T20:26:14.531691Z","iopub.status.idle":"2021-06-26T20:26:14.999223Z","shell.execute_reply.started":"2021-06-26T20:26:14.531634Z","shell.execute_reply":"2021-06-26T20:26:14.998262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"notice it can be broken down more in the address section","metadata":{}},{"cell_type":"code","source":"ex_output[\"address\"]","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:15.000856Z","iopub.execute_input":"2021-06-26T20:26:15.001166Z","iopub.status.idle":"2021-06-26T20:26:15.006896Z","shell.execute_reply.started":"2021-06-26T20:26:15.001138Z","shell.execute_reply":"2021-06-26T20:26:15.006045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Mini Dataset For A Single Path\n\nSince every call of my geocoder function takes quite a while to run, the example below uses only every 5 data points in a single path.\n\nHere are the features I will be taking a closer look at: place id (numerical id of the place), lat or adjusted lat (a lat value essentially generated through snap to structure), lon or adjusted lon (a lon value essentially generated through snap to structure), road (the name of the road the point falls on), man_made (name of a man made structure the point falls on)","metadata":{}},{"cell_type":"code","source":"ex_base = base_test[base_test.phone == '2021-04-02-US-SJC-1_Pixel4']\nex_base = ex_base[::5]\nex_base.reset_index(drop=True, inplace=True)\nex_base.tail(2)","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:15.008069Z","iopub.execute_input":"2021-06-26T20:26:15.008359Z","iopub.status.idle":"2021-06-26T20:26:15.054029Z","shell.execute_reply.started":"2021-06-26T20:26:15.008333Z","shell.execute_reply":"2021-06-26T20:26:15.053238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_data = pd.DataFrame(columns=[\"place_id\", \"adj_lat\", \"adj_lon\", \"road\", \"man_made\"])\n\nfor i in tqdm(range(len(ex_base))): \n    data = geocoder(ex_base.iloc[i])\n    try:\n        man_made = data[\"address\"][\"man_made\"]\n    except:\n        man_made = 'nan'\n    ex_data = ex_data.append({'place_id': data[\"place_id\"],\n                              'adj_lat': data[\"lat\"],\n                              'adj_lon': data[\"lon\"],\n                              'road': data[\"address\"][\"road\"],\n                              'man_made': man_made}, \n                              ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:26:15.055365Z","iopub.execute_input":"2021-06-26T20:26:15.055749Z","iopub.status.idle":"2021-06-26T20:30:11.506132Z","shell.execute_reply.started":"2021-06-26T20:26:15.055697Z","shell.execute_reply":"2021-06-26T20:30:11.505038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ex_data.apply(pd.to_numeric, errors='ignore')\nex_data = ex_data.join(ex_base)\nex_data = ex_data.apply(pd.to_numeric, errors='ignore')\nex_data.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:30:11.507398Z","iopub.execute_input":"2021-06-26T20:30:11.507674Z","iopub.status.idle":"2021-06-26T20:30:11.538714Z","shell.execute_reply.started":"2021-06-26T20:30:11.507643Z","shell.execute_reply":"2021-06-26T20:30:11.537708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### EDA of Mini Dataset","metadata":{}},{"cell_type":"code","source":"print('place id')\nprint(ex_data.place_id.value_counts(), '\\n')\n\nprint('road:')\nprint(ex_data.road.value_counts(), '\\n')\n\nprint('man_made:')\nprint(ex_data.man_made.value_counts(), '\\n')\n\nprint('adj_lat:')\nprint('mean', np.mean(ex_data.adj_lat))\nprint('median', np.median(ex_data.adj_lat))\nprint('mode', stats.mode(ex_data.adj_lat)[0][0], '\\n')\n\nprint('adj_lon:')\nprint('mean', np.mean(ex_data.adj_lon))\nprint('median', np.median(ex_data.adj_lon))\nprint('mode', stats.mode(ex_data.adj_lon)[0][0])","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:30:11.539994Z","iopub.execute_input":"2021-06-26T20:30:11.540283Z","iopub.status.idle":"2021-06-26T20:30:11.562291Z","shell.execute_reply.started":"2021-06-26T20:30:11.540256Z","shell.execute_reply":"2021-06-26T20:30:11.561101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"place_ids = list()\nplace_id_values = list()\nother_count = 0\nfor place_id in ex_data.place_id.value_counts().index:\n    if(ex_data.place_id.value_counts()[place_id] > 6):\n        place_ids.append(place_id)\n        place_id_values.append(ex_data.place_id.value_counts()[place_id])\n    else:\n        other_count+=ex_data.place_id.value_counts()[place_id]\nplace_ids.append(\"other\")\nplace_id_values.append(other_count)\n\nplt.pie(place_id_values)\nplt.legend(place_ids, bbox_to_anchor=(1.2,0.5), loc=\"center right\", fontsize=10, \n           bbox_transform=plt.gcf().transFigure)\nplt.title(\"Place ID\")\nplt.plot()","metadata":{"execution":{"iopub.status.busy":"2021-06-26T21:00:11.640931Z","iopub.execute_input":"2021-06-26T21:00:11.641496Z","iopub.status.idle":"2021-06-26T21:00:11.984007Z","shell.execute_reply.started":"2021-06-26T21:00:11.641462Z","shell.execute_reply":"2021-06-26T21:00:11.983342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.pie(ex_data.road.value_counts())\nplt.legend(ex_data.road.value_counts().index, bbox_to_anchor=(1.2,0.5), loc=\"center right\", fontsize=10, \n           bbox_transform=plt.gcf().transFigure)\nplt.title(\"Road\")\nplt.plot()","metadata":{"execution":{"iopub.status.busy":"2021-06-26T20:50:14.642814Z","iopub.execute_input":"2021-06-26T20:50:14.643181Z","iopub.status.idle":"2021-06-26T20:50:14.905607Z","shell.execute_reply.started":"2021-06-26T20:50:14.643148Z","shell.execute_reply":"2021-06-26T20:50:14.904669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.pie(ex_data.man_made.value_counts())\nplt.legend(ex_data.man_made.value_counts().index, bbox_to_anchor=(1.4,0.5), loc=\"center right\", fontsize=10, \n           bbox_transform=plt.gcf().transFigure)\nplt.title(\"Man Made\")\nplt.plot()","metadata":{"execution":{"iopub.status.busy":"2021-06-26T21:01:14.028537Z","iopub.execute_input":"2021-06-26T21:01:14.028921Z","iopub.status.idle":"2021-06-26T21:01:14.204661Z","shell.execute_reply.started":"2021-06-26T21:01:14.028886Z","shell.execute_reply":"2021-06-26T21:01:14.203609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is what the path looks like on a map.","metadata":{}},{"cell_type":"code","source":"fig = px.scatter_mapbox(ex_data,\n\n                    # Here, plotly gets, (x,y) coordinates\n                    lat=\"latDeg\",\n                    lon=\"lngDeg\",\n                    text='phoneName',\n\n                    #Here, plotly detects color of series\n                    color=\"collectionName\",\n                    labels=\"collectionName\",\n\n                    zoom=12,\n                    center={\"lat\":np.mean(ex_data.latDeg), \"lon\":np.mean(ex_data.lngDeg)},\n                    height=600,\n                    width=800)\nfig.update_layout(mapbox_style='stamen-terrain')\nfig.update_layout(margin={\"r\": 0, \"t\": 0, \"l\": 0, \"b\": 0})\nfig.update_layout(title_text=\"Original\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-26T21:03:03.710989Z","iopub.execute_input":"2021-06-26T21:03:03.711348Z","iopub.status.idle":"2021-06-26T21:03:03.780500Z","shell.execute_reply.started":"2021-06-26T21:03:03.711319Z","shell.execute_reply":"2021-06-26T21:03:03.779787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is a comparison of the lat/lon path in the test file vs the adjusted results. Unfortunately the adjusted results are subpar ","metadata":{}},{"cell_type":"code","source":"plt.subplots(nrows=1, ncols=2, figsize=(11, 5))\nplt.subplot(1,2,1)\nplt.title('original')\nplt.xlabel('lon')\nplt.ylabel('lat')\nplt.plot(ex_data.latDeg, ex_data.lngDeg, color='blue')\nplt.subplot(1,2,2)\nplt.title('rev geocoded')\nplt.xlabel('lon')\nplt.ylabel('lat')\nplt.plot(ex_data.adj_lat, ex_data.adj_lon, color='red')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-26T21:02:10.830095Z","iopub.execute_input":"2021-06-26T21:02:10.830472Z","iopub.status.idle":"2021-06-26T21:02:11.115135Z","shell.execute_reply.started":"2021-06-26T21:02:10.830438Z","shell.execute_reply":"2021-06-26T21:02:11.114104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nReverse Geocoders can definitely be useful to generate more data that can be used as model features or inputs to post processing.\n\nSome good features to take a look at are place id, road, man made, bounding box, postcode, commercial, and more.\n\nThe snap to structure with reverse geocoders is not as good as the original; however, we can use other external data tools such as openstreetmap.org","metadata":{}}]}