{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Let's visualize dataset to understand! πŸš™δΈ‰\n\nIt may be difficult to imagine what kind of data we are dealing with in the first place. We have focused on the visualization of ground truth of smartphone tracks to help you understand the data better. As we work through the competition, we will want to know how accurately our solution reproduces the trackings, and I also hope this will be helpful in visualizing them.\n\n## Note\n\nCells where you can change the analysis target by changing the values, I'll mark with orange sentence like this!!!\n\n## Contents\n\n* [Background](#1)\n * [Goal](#1-1)\n * [About dataset](#1-2)\n* [Data overview](#2)\n * [Load data and see overview](#2-1)\n * [How to check track in detail?](#2-2)\n * [How to check large amounts of tracks?](#2-3)\n * [How to check tracks in animation?](#2-4)\n * [How to load data from .nmea file](#2-5)\n* [Appendix](#3)\n * [Distribution of rollDegs of phones](#3-1)\n * [Which collection contains which phone](#3-2)","metadata":{}},{"cell_type":"code","source":"!pip install pynmea2\n\nimport glob\nimport itertools\nimport json\nimport os\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport geopandas as gpd\nfrom geopandas import GeoDataFrame\nimport geoplot as gplt\nfrom IPython.display import Video\nfrom matplotlib import animation\nimport matplotlib.pyplot as plt\nimport numpy as np \nimport pandas as pd\nimport plotly.express as px\nimport pynmea2\nimport requests\nimport seaborn\nfrom shapely.geometry import Point, shape\nimport shapely.wkt\n\n%matplotlib inline\n\nDATA_PATH = \"../input/google-smartphone-decimeter-challenge/\"","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-23T08:09:55.071137Z","iopub.execute_input":"2021-05-23T08:09:55.071601Z","iopub.status.idle":"2021-05-23T08:10:02.957829Z","shell.execute_reply.started":"2021-05-23T08:09:55.071566Z","shell.execute_reply":"2021-05-23T08:10:02.956444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Background\n\n\n## Goal\n\nThe goal of this competition is to compute location (latDeg and lngDeg) down to decimeter or even centimeter resolution, if possible for each test phone and time.\n\nOur submission file is like this.","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv(DATA_PATH + \"sample_submission.csv\")\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:14.224315Z","iopub.execute_input":"2021-05-23T06:45:14.224716Z","iopub.status.idle":"2021-05-23T06:45:14.429180Z","shell.execute_reply.started":"2021-05-23T06:45:14.224673Z","shell.execute_reply":"2021-05-23T06:45:14.428073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"latDeg and lngDeg are our target.\n\nWe can use GPS tracking data and variety of sensor data to improve our solution. ","metadata":{}},{"cell_type":"markdown","source":"\n## About dataset \n\nGoogle releases 60+ datasets collected from phones in the Android GPS team, together with corrections from SwiftNavigation Inc. and Verizon Inc. These datasets were collected on highways in the US San Francisco Bay Area in the summer of 2020. We can see the video for dataset.","metadata":{}},{"cell_type":"code","source":"!cp \"../input/android-smartphones-high-accuracy-datasets/ION GNSS 2020 Presentation (Michael Fu).mp4\" ION_GNSS_2020_Presentation.mp4","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:45:14.431012Z","iopub.execute_input":"2021-05-23T06:45:14.431294Z","iopub.status.idle":"2021-05-23T06:45:17.594978Z","shell.execute_reply.started":"2021-05-23T06:45:14.431267Z","shell.execute_reply":"2021-05-23T06:45:17.593627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Video(\"ION_GNSS_2020_Presentation.mp4\", width=640, height=640)","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:17.597268Z","iopub.execute_input":"2021-05-23T06:45:17.597675Z","iopub.status.idle":"2021-05-23T06:45:17.605582Z","shell.execute_reply.started":"2021-05-23T06:45:17.597627Z","shell.execute_reply":"2021-05-23T06:45:17.604566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are given data from actual runs with android devices installed in cars, see following. ","metadata":{}},{"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/tasotasoso/kaggle_media/main/Android_smartphones_high_accuracy_GNSS_datasets/fig3_fig4.JPG)\n\nThe figures come from Fu, Guoyu (Michael), Khider, Mohammed, van Diggelen, Frank, \"Android Raw GNSS Measurement Datasets for Precise Positioning,\" Proceedings of the 33rd International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2020), September 2020, pp. 1925-1937.\n[https://doi.org/10.33012/2020.17628](https://www.ion.org/publications/abstract.cfm?articleID=17628)\n\nWe can see more detail of data collection process at [Android smartphones high accuracy GNSS datasets](https://www.kaggle.com/google/android-smartphones-high-accuracy-datasets).","metadata":{}},{"cell_type":"markdown","source":" Devices can be one thing or multiple things. Data collection trials are separated as collectionName like this.","metadata":{}},{"cell_type":"code","source":"!ls ../input/google-smartphone-decimeter-challenge/train","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:17.607068Z","iopub.execute_input":"2021-05-23T06:45:17.607568Z","iopub.status.idle":"2021-05-23T06:45:18.342213Z","shell.execute_reply.started":"2021-05-23T06:45:17.607526Z","shell.execute_reply":"2021-05-23T06:45:18.341095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Under each collectionName, the data of the device is stored.","metadata":{}},{"cell_type":"code","source":"!ls ../input/google-smartphone-decimeter-challenge/train/2020-05-14-US-MTV-1/","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:18.343610Z","iopub.execute_input":"2021-05-23T06:45:18.343895Z","iopub.status.idle":"2021-05-23T06:45:19.083704Z","shell.execute_reply.started":"2021-05-23T06:45:18.343865Z","shell.execute_reply":"2021-05-23T06:45:19.082412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In addition, the data collected from each device, groundtruth, and supplemental data are stored under it.","metadata":{}},{"cell_type":"code","source":"!ls ../input/google-smartphone-decimeter-challenge/train/2020-05-14-US-MTV-1/Pixel4","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:19.085639Z","iopub.execute_input":"2021-05-23T06:45:19.086098Z","iopub.status.idle":"2021-05-23T06:45:19.823601Z","shell.execute_reply.started":"2021-05-23T06:45:19.086046Z","shell.execute_reply":"2021-05-23T06:45:19.822658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls ../input/google-smartphone-decimeter-challenge/train/2020-05-14-US-MTV-1/Pixel4/supplemental","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:19.828297Z","iopub.execute_input":"2021-05-23T06:45:19.829236Z","iopub.status.idle":"2021-05-23T06:45:20.564750Z","shell.execute_reply.started":"2021-05-23T06:45:19.829111Z","shell.execute_reply":"2021-05-23T06:45:20.563491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The supplemental data contains the raw data that was measured, and I'll show way to read nmea.","metadata":{}},{"cell_type":"markdown","source":"\n# Data overview\n\n\n## Load data and see overview\n\nYou can use [train/test]/[drive_id]/[phone_name]/[phone_name]_derived.csv as organized data. And also we can use ground_truth.csv as reference. Let's open one of the files and see what it contains.","metadata":{}},{"cell_type":"code","source":"!ls ../input/google-smartphone-decimeter-challenge/train/2020-05-14-US-MTV-1/","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:20.567114Z","iopub.execute_input":"2021-05-23T06:45:20.567437Z","iopub.status.idle":"2021-05-23T06:45:21.296641Z","shell.execute_reply.started":"2021-05-23T06:45:20.567399Z","shell.execute_reply":"2021-05-23T06:45:21.295431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_trail = pd.read_csv(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/Pixel4_derived.csv\")\ndf_sample_trail_gt = pd.read_csv(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/ground_truth.csv\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:45:21.300923Z","iopub.execute_input":"2021-05-23T06:45:21.301290Z","iopub.status.idle":"2021-05-23T06:45:21.655875Z","shell.execute_reply.started":"2021-05-23T06:45:21.301251Z","shell.execute_reply":"2021-05-23T06:45:21.654335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_trail.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:21.657395Z","iopub.execute_input":"2021-05-23T06:45:21.657713Z","iopub.status.idle":"2021-05-23T06:45:21.688303Z","shell.execute_reply.started":"2021-05-23T06:45:21.657682Z","shell.execute_reply":"2021-05-23T06:45:21.687196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_trail.columns","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:21.689755Z","iopub.execute_input":"2021-05-23T06:45:21.690377Z","iopub.status.idle":"2021-05-23T06:45:21.702759Z","shell.execute_reply.started":"2021-05-23T06:45:21.690332Z","shell.execute_reply":"2021-05-23T06:45:21.701376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_trail_gt.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:21.704569Z","iopub.execute_input":"2021-05-23T06:45:21.705006Z","iopub.status.idle":"2021-05-23T06:45:21.728249Z","shell.execute_reply.started":"2021-05-23T06:45:21.704963Z","shell.execute_reply":"2021-05-23T06:45:21.726971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_trail_gt.columns","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:21.729680Z","iopub.execute_input":"2021-05-23T06:45:21.730003Z","iopub.status.idle":"2021-05-23T06:45:21.741388Z","shell.execute_reply.started":"2021-05-23T06:45:21.729973Z","shell.execute_reply":"2021-05-23T06:45:21.740112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## How to check track in detail?\n\nWe can use plotly to see our model or ground truth like this. To see trafic, you should adjust map centor and scale.\n\n","metadata":{}},{"cell_type":"code","source":"def visualize_trafic(df, center, zoom=9):\n fig = px.scatter_mapbox(df,\n \n # Here, plotly gets, (x,y) coordinates\n lat=\"latDeg\",\n lon=\"lngDeg\",\n \n #Here, plotly detects color of series\n color=\"phoneName\",\n labels=\"phoneName\",\n \n zoom=zoom,\n center=center,\n height=600,\n width=800)\n fig.update_layout(mapbox_style='stamen-terrain')\n fig.update_layout(margin={\"r\": 0, \"t\": 0, \"l\": 0, \"b\": 0})\n fig.update_layout(title_text=\"GPS trafic\")\n fig.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:21.743058Z","iopub.execute_input":"2021-05-23T06:45:21.744017Z","iopub.status.idle":"2021-05-23T06:45:21.751736Z","shell.execute_reply.started":"2021-05-23T06:45:21.743968Z","shell.execute_reply":"2021-05-23T06:45:21.750983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ↓↓↓ If you want to check other track, you should load the data in below cell.","metadata":{}},{"cell_type":"code","source":"# I will reload dataframe so that it is easy to look at other data.\ndf_sample_trail_gt = pd.read_csv(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/ground_truth.csv\")\n\ncenter = {\"lat\":37.423576, \"lon\":-122.094132}\nvisualize_trafic(df_sample_trail_gt, center)","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:21.753057Z","iopub.execute_input":"2021-05-23T06:45:21.753680Z","iopub.status.idle":"2021-05-23T06:45:22.872149Z","shell.execute_reply.started":"2021-05-23T06:45:21.753636Z","shell.execute_reply":"2021-05-23T06:45:22.871115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data which have same collectionName seems have same ground truth.\n### ↓↓↓ If you want to check other track, you should load the data in below cell.","metadata":{}},{"cell_type":"code","source":"# I will reload dataframe so that it is easy to look at other data.\ndf_sample_trail_gt = pd.read_csv(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/ground_truth.csv\")\ndf_sample_trail_gt2 = pd.read_csv(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4XLModded/ground_truth.csv\")\n\n# Since plotly looks at the phoneName of the dataframe,\n# you can visualize multiple series of data by simply concatting dataframes.\ndf_sample_trail_gt3 = pd.concat([df_sample_trail_gt, df_sample_trail_gt2])\n\ncenter = {\"lat\":37.423576, \"lon\":-122.094132}\nvisualize_trafic(df_sample_trail_gt3, center)","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:22.873597Z","iopub.execute_input":"2021-05-23T06:45:22.874016Z","iopub.status.idle":"2021-05-23T06:45:22.986571Z","shell.execute_reply.started":"2021-05-23T06:45:22.873976Z","shell.execute_reply":"2021-05-23T06:45:22.985279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## How to check large amounts of tracks?\n\nEarlier we saw how to use plotly to map data on OpenStreetMap. This time, since there is a certain amount of tracking data in the train data alone, I will also show you how to use geopandas to get a quick overview as a regular diagram.","metadata":{}},{"cell_type":"markdown","source":"From here on, the cells will be hidden for a while, because the procedure is necessary for visualization and we will get tired of following everything. If you have interest, please open and check them accordingly.\n\nFirst, I'll download shape file lof bayarea.","metadata":{}},{"cell_type":"code","source":"#Download geojson file of US San Francisco Bay Area.\nr = requests.get(\"https://data.sfgov.org/api/views/wamw-vt4s/rows.json?accessType=DOWNLOAD\")\nr.raise_for_status()\n\n#get geojson from response\ndata = r.json()\n\n#get polygons that represents San Francisco Bay Area.\nshapes = []\nfor d in data[\"data\"]:\n shapes.append(shapely.wkt.loads(d[8]))\n \n#Convert list of porygons to geopandas dataframe.\ngdf_bayarea = pd.DataFrame()\n\n#I'll use only 6 and 7th object.\nfor shp in shapes[5:7]:\n tmp = pd.DataFrame(shp, columns=[\"geometry\"])\n gdf_bayarea = pd.concat([gdf_bayarea, tmp])\ngdf_bayarea = GeoDataFrame(gdf_bayarea)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:45:22.988194Z","iopub.execute_input":"2021-05-23T06:45:22.988587Z","iopub.status.idle":"2021-05-23T06:45:24.497692Z","shell.execute_reply.started":"2021-05-23T06:45:22.988548Z","shell.execute_reply":"2021-05-23T06:45:24.496741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each collectionNames, read the ground truth files in format that is convenient for visualization. At this time, we have already converted it to geopandas dataframe.","metadata":{}},{"cell_type":"code","source":"collectionNames = [item.split(\"/\")[-1] for item in glob.glob(\"../input/google-smartphone-decimeter-challenge/train/*\")]\n\ngdfs = []\nfor collectionName in collectionNames:\n gdfs_each_collectionName = []\n csv_paths = glob.glob(f\"../input/google-smartphone-decimeter-challenge/train/{collectionName}/*/ground_truth.csv\")\n for csv_path in csv_paths:\n df_gt = pd.read_csv(csv_path)\n df_gt[\"geometry\"] = [Point(lngDeg, latDeg) for lngDeg, latDeg in zip(df_gt[\"lngDeg\"], df_gt[\"latDeg\"])]\n gdfs_each_collectionName.append(GeoDataFrame(df_gt))\n gdfs.append(gdfs_each_collectionName)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:45:24.499120Z","iopub.execute_input":"2021-05-23T06:45:24.499536Z","iopub.status.idle":"2021-05-23T06:45:27.660050Z","shell.execute_reply.started":"2021-05-23T06:45:24.499494Z","shell.execute_reply":"2021-05-23T06:45:27.658885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colors = ['blue', 'green', 'purple', 'orange']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:45:27.661379Z","iopub.execute_input":"2021-05-23T06:45:27.661657Z","iopub.status.idle":"2021-05-23T06:45:27.666839Z","shell.execute_reply.started":"2021-05-23T06:45:27.661631Z","shell.execute_reply":"2021-05-23T06:45:27.665925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's visualize tracks. Some of them were too small to be seen when projected on the map, so we put them side by side with the ones that are just routes.","metadata":{}},{"cell_type":"code","source":"for collectionName, gdfs_each_collectionName in zip(collectionNames, gdfs):\n fig, axs = plt.subplots(1, 2, figsize=(15, 5))\n gdf_bayarea.plot(figsize=(10,10), color='none', edgecolor='gray', zorder=3, ax=axs[0])\n for i, gdf in enumerate(gdfs_each_collectionName):\n g1 = gdf.plot(color=colors[i], ax=axs[0])\n g1.set_title(f\"Phone track of {collectionName} with map\")\n g2 = gdf.plot(color=colors[i], ax=axs[1])\n g2.set_title(f\"Phone track of {collectionName}\")","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:45:27.668147Z","iopub.execute_input":"2021-05-23T06:45:27.668438Z","iopub.status.idle":"2021-05-23T06:46:07.570327Z","shell.execute_reply.started":"2021-05-23T06:45:27.668408Z","shell.execute_reply":"2021-05-23T06:46:07.569111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are several tracks that have the same form of data with different collectionName. It is easy to understand the positional relationship by overlapping them. There are two roads extending from the northwest to the southeast, and they seem to run along those roads all the time, or occasionally go off those roads. The tracks wandering around the grid-like paths seem to be collected farther southeast than those paths.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 5))\n\nfor collectionName, gdfs_each_collectionName in zip(collectionNames, gdfs): \n for i, gdf in enumerate(gdfs_each_collectionName):\n gdf.plot(color=colors[i], ax=ax, markersize=5, alpha=0.5)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:46:07.571736Z","iopub.execute_input":"2021-05-23T06:46:07.572060Z","iopub.status.idle":"2021-05-23T06:46:33.090792Z","shell.execute_reply.started":"2021-05-23T06:46:07.572031Z","shell.execute_reply":"2021-05-23T06:46:33.089637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In geopandas, it's easy to see where they are in relation to each other, but it's hard to see the details and geographic information, so let's look at them with plotly as well.","metadata":{}},{"cell_type":"code","source":"all_tracks = pd.DataFrame()\n\nfor collectionName, gdfs_each_collectionName in zip(collectionNames, gdfs): \n for i, gdf in enumerate(gdfs_each_collectionName):\n all_tracks = pd.concat([all_tracks, gdf])\n # Tracks they have same collectionName is also same\n break\n \nfig = px.scatter_mapbox(all_tracks,\n \n # Here, plotly gets, (x,y) coordinates\n lat=\"latDeg\",\n lon=\"lngDeg\",\n \n #Here, plotly detects color of series\n color=\"collectionName\",\n labels=\"collectionName\",\n \n zoom=9,\n center={\"lat\":37.423576, \"lon\":-122.094132},\n height=600,\n width=800)\nfig.update_layout(mapbox_style='stamen-terrain')\nfig.update_layout(margin={\"r\": 0, \"t\": 0, \"l\": 0, \"b\": 0})\nfig.update_layout(title_text=\"GPS trafic\")\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:46:33.094662Z","iopub.execute_input":"2021-05-23T06:46:33.095054Z","iopub.status.idle":"2021-05-23T06:46:33.749430Z","shell.execute_reply.started":"2021-05-23T06:46:33.095016Z","shell.execute_reply":"2021-05-23T06:46:33.748356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## How to check tracks in animation?\n\nI am sure that you all would like to see animations of what the data was driving at. I have also prepared code to check the x, y coordinate movement in gif, so try it out. \n\nThe process takes some minutes (about 2 or 3 minutes for my implement).\n\nNote: The x, y coordinates have been thinned out (to 1/10) because the processing uses much memory. You can adjust them if necessary.","metadata":{}},{"cell_type":"code","source":"def create_gif_track(df, git_path):\n \"\"\" Create git animation of phone track.\n \"\"\"\n\n fig, ax = plt.subplots()\n\n imgs = []\n df[\"geometry\"] = [Point(lngDeg, latDeg) for lngDeg, latDeg in zip(df[\"lngDeg\"], df[\"latDeg\"])]\n gdf = GeoDataFrame(df)\n gdf.plot(color=\"lightskyblue\", ax=ax)\n \n # Here, (x,y) coordinates are thinned out!!!\n for i in range(0, len(gdf), 10):\n # plot data\n p = ax.plot(gdf.iloc[i][\"lngDeg\"], gdf.iloc[i][\"latDeg\"], \n color = 'dodgerblue', marker = 'o', markersize = 8)\n imgs.append(p)\n\n # Create animation & save it\n ani = animation.ArtistAnimation(fig, imgs, interval=200)\n ani.save(git_path, writer='imagemagick', dpi = 300)\n \n\ndef create_gif_track_on_map(df, gdf_map, git_path):\n \"\"\" Create git animation of phone track on bayarea map.\n \"\"\"\n\n fig, ax = plt.subplots()\n df[\"geometry\"] = [Point(lngDeg, latDeg) for lngDeg, latDeg in zip(df[\"lngDeg\"], df[\"latDeg\"])]\n gdf = GeoDataFrame(df)\n gdf.plot(color=\"lightskyblue\", ax=ax)\n imgs = [] \n gdf_map.plot(color='none', edgecolor='gray', zorder=3, ax=ax)\n \n # Here, (x,y) coordinates are thinned out!!!\n for i in range(0, len(gdf), 10):\n # plot data on map\n p = ax.plot(gdf.iloc[i][\"lngDeg\"], gdf.iloc[i][\"latDeg\"], \n color = 'dodgerblue', marker = 'o', markersize = 8)\n imgs.append(p)\n\n # Create animation & save it\n ani = animation.ArtistAnimation(fig, imgs, interval=200)\n ani.save(git_path, writer='imagemagick', dpi = 300)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:46:33.751212Z","iopub.execute_input":"2021-05-23T06:46:33.751517Z","iopub.status.idle":"2021-05-23T06:46:33.764775Z","shell.execute_reply.started":"2021-05-23T06:46:33.751484Z","shell.execute_reply":"2021-05-23T06:46:33.763659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ↓↓↓ You can change the first argment for dataframe of visualize target and the second argment for gif file to save.","metadata":{}},{"cell_type":"code","source":"%%time\n\ndf_2021_04_29_US_SJC_2_Pixel4_gt = pd.read_csv(DATA_PATH + \"train/2021-04-29-US-SJC-2/Pixel4/ground_truth.csv\")\n\ncreate_gif_track(df_2021_04_29_US_SJC_2_Pixel4_gt, \"./2021_04_29_US_SJC_2_Pixel4_gt.gif\")","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:46:33.766162Z","iopub.execute_input":"2021-05-23T06:46:33.766473Z","iopub.status.idle":"2021-05-23T06:49:54.940078Z","shell.execute_reply.started":"2021-05-23T06:46:33.766441Z","shell.execute_reply":"2021-05-23T06:49:54.939055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ↓↓↓ You can change the first argment for dataframe of visualize target and the third argment for gif file to save.","metadata":{}},{"cell_type":"code","source":"%%time\n\ndf_2020_05_14_US_MTV_1_Pixel4_gt = pd.read_csv(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/ground_truth.csv\")\n\ncreate_gif_track_on_map(df_2020_05_14_US_MTV_1_Pixel4_gt, gdf_bayarea, \"./2020_05_14_US_MTV_1_Pixel4_gt.gif\")","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:49:54.941704Z","iopub.execute_input":"2021-05-23T06:49:54.942058Z","iopub.status.idle":"2021-05-23T06:52:30.580247Z","shell.execute_reply.started":"2021-05-23T06:49:54.942024Z","shell.execute_reply":"2021-05-23T06:52:30.579139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Gif animation of ../input/google-smartphone-decimeter-challenge/train/2021_04_29_US_SJC_2/Pixel4/ground_truth.csv\n\n![2021_04_29_US_SJC_2_Pixel4_gt.gif](./2021_04_29_US_SJC_2_Pixel4_gt.gif)\n\n
Info: To see another gif animation you create, you can change the path of above line of this line in this markdown cell.
","metadata":{}},{"cell_type":"markdown","source":"### Gif animation of ../input/google-smartphone-decimeter-challenge/train/2020-05-14-US-MTV-1/Pixel4/ground_truth.csv\n\n![2020_05_14_US_MTV_1_Pixel4_gt.gif](./2020_05_14_US_MTV_1_Pixel4_gt.gif)\n\n
Info: To see another gif animation you create, you can change the path of above line of this line in this markdown cell.
","metadata":{}},{"cell_type":"markdown","source":"\n## How to load data from .nmea file\n\nWe can also use supplemental data. Now we'll see how to load .nmea file.\n\nNMEA (National Marine Electronics Association) format is GPS file format. To extract data from .nmea file, we can use [pynmea2](https://github.com/Knio/pynmea2) library. In the NMEA file, each location is described jointly by a pair of GGA and RMC sentences, as depicted in following Figures 5 and 6.\n\n![GGA & RMC format](https://raw.githubusercontent.com/tasotasoso/kaggle_media/main/Android_smartphones_high_accuracy_GNSS_datasets/fig5_fig6.JPG)\n\nThe figures come from Fu, Guoyu (Michael), Khider, Mohammed, van Diggelen, Frank, \"Android Raw GNSS Measurement Datasets for Precise Positioning,\" Proceedings of the 33rd International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS+ 2020), September 2020, pp. 1925-1937.\n[https://doi.org/10.33012/2020.17628](https://www.ion.org/publications/abstract.cfm?articleID=17628)","metadata":{}},{"cell_type":"code","source":"with open(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/supplemental/SPAN_Pixel4_10Hz.nmea\", encoding='utf-8') as nmea_f:\n for line in nmea_f.readlines():\n try:\n msg = pynmea2.parse(line)\n break\n except pynmea2.ParseError as e:\n pass\n \nprint(repr(msg))\nprint(\"---------------------------------\")\nprint(\"timestamp:\", msg.timestamp)\nprint(\"lat:\", msg.lat)\nprint(\"lat_dir:\", msg.lat_dir)\nprint(\"lon:\", msg.lon)\nprint(\"lon_dir:\", msg.lon_dir)\nprint(\"gps_qual:\", msg.gps_qual)\nprint(\"num_sats:\", msg.num_sats)\nprint(\"horizontal_dil:\", msg.horizontal_dil)\nprint(\"altitude:\", msg.altitude)\nprint(\"altitude_units:\", msg.altitude_units)\nprint(\"geo_sep:\", msg.geo_sep)\nprint(\"geo_sep_units:\", msg.geo_sep_units)\nprint(\"age_gps_data:\", msg.age_gps_data)\nprint(\"ref_station_id:\", msg.ref_station_id)\nprint(\"---------------------------------\")","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:52:30.582102Z","iopub.execute_input":"2021-05-23T06:52:30.582562Z","iopub.status.idle":"2021-05-23T06:52:30.666275Z","shell.execute_reply.started":"2021-05-23T06:52:30.582512Z","shell.execute_reply":"2021-05-23T06:52:30.665363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_nmea_file(nmea_path):\n \"\"\"Convert GGA sentences of nmea files to pandas dataframe.\n \"\"\"\n \n gga_lines = []\n with open(nmea_path, encoding='utf-8') as nmea_f:\n for line in nmea_f.readlines():\n try:\n msg = pynmea2.parse(line)\n gga_lines.append(msg)\n except pynmea2.ParseError as e:\n pass\n \n \n return pd.DataFrame(gga_lines)","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:52:30.667718Z","iopub.execute_input":"2021-05-23T06:52:30.668073Z","iopub.status.idle":"2021-05-23T06:52:30.673753Z","shell.execute_reply.started":"2021-05-23T06:52:30.668043Z","shell.execute_reply":"2021-05-23T06:52:30.672731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nmea_sample = load_nmea_file(DATA_PATH + \"train/2020-05-14-US-MTV-1/Pixel4/supplemental/SPAN_Pixel4_10Hz.nmea\")","metadata":{"execution":{"iopub.status.busy":"2021-05-23T06:52:30.674814Z","iopub.execute_input":"2021-05-23T06:52:30.675111Z","iopub.status.idle":"2021-05-23T06:52:31.382015Z","shell.execute_reply.started":"2021-05-23T06:52:30.675083Z","shell.execute_reply":"2021-05-23T06:52:31.380859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Appendix\n\nAdditional analysis of personal interest is posted here for reference.\n\n\n## Distribution of rollDegs of phones\n\nFrom the fact that the phones are standing up in the car like image above, and from the definition of rollDeg, we should be able to tell which direction the phone traveled by checking rollDeg.\n\nFrom [Data](https://www.kaggle.com/c/google-smartphone-decimeter-challenge/data), \n\n> rollDeg - Roll, angle of rotation about the y axis. This value represents the angle between a plane perpendicular to the device's screen and a plane perpendicular to the ground.\n\nI am interested in this point because I guess direction phone rotate will affect where go next. Some devices are able to take rollDeg and some don't. I am particularly interested in whether the data is available for all orientations. Unfortunately, rollDeg is only available for logs collected after March 2021.","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/carlmcbrideellis/google-smartphone-decimeter-eda\n\ndef gnss_log_to_dataframes(path):\n print('Loading ' + path, flush=True)\n gnss_section_names = {'Raw','UncalAccel', 'UncalGyro', 'UncalMag', 'Fix', 'Status', 'OrientationDeg'}\n with open(path) as f_open:\n datalines = f_open.readlines()\n\n datas = {k: [] for k in gnss_section_names}\n gnss_map = {k: [] for k in gnss_section_names}\n for dataline in datalines:\n is_header = dataline.startswith('#')\n dataline = dataline.strip('#').strip().split(',')\n # skip over notes, version numbers, etc\n if is_header and dataline[0] in gnss_section_names:\n gnss_map[dataline[0]] = dataline[1:]\n elif not is_header:\n datas[dataline[0]].append(dataline[1:])\n\n results = dict()\n for k, v in datas.items():\n results[k] = pd.DataFrame(v, columns=gnss_map[k])\n # pandas doesn't properly infer types from these lists by default\n for k, df in results.items():\n for col in df.columns:\n if col == 'CodeType':\n continue\n results[k][col] = pd.to_numeric(results[k][col])\n\n return results","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T06:52:31.383373Z","iopub.execute_input":"2021-05-23T06:52:31.383669Z","iopub.status.idle":"2021-05-23T06:52:31.393485Z","shell.execute_reply.started":"2021-05-23T06:52:31.383641Z","shell.execute_reply":"2021-05-23T06:52:31.392574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gnsslog_paths = glob.glob(f\"../input/google-smartphone-decimeter-challenge/train/*\")\n\ndfs = []\nfor collectionName in collectionNames:\n dfs_each_collectionName = []\n gnsslog_paths = glob.glob(f\"../input/google-smartphone-decimeter-challenge/train/{collectionName}/*/*_GnssLog.txt\")\n for gnsslog_path in gnsslog_paths:\n try:\n orientation_degs = gnss_log_to_dataframes(gnsslog_path)[\"OrientationDeg\"]\n except:\n print(f\"pass {gnsslog_path}\")\n dfs_each_collectionName.append(pd.DataFrame())\n continue\n dfs_each_collectionName.append(orientation_degs)\n dfs.append(dfs_each_collectionName)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-05-23T06:52:31.394686Z","iopub.execute_input":"2021-05-23T06:52:31.395013Z","iopub.status.idle":"2021-05-23T07:09:32.086752Z","shell.execute_reply.started":"2021-05-23T06:52:31.394984Z","shell.execute_reply":"2021-05-23T07:09:32.085794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(len(collectionNames)):\n fig = plt.figure(figsize=(10, 5))\n axes0 = fig.add_subplot(121)\n axes1 = fig.add_subplot(122, projection='polar')\n axes1.set_theta_zero_location('N')\n axes1.set_theta_direction(-1)\n \n collectionName = collectionNames[i]\n gdfs_each_collectionName = gdfs[i]\n dfs_each_collectionName = dfs[i]\n \n for j in range(len(gdfs_each_collectionName)):\n g1 = gdfs_each_collectionName[j].plot(color=colors[j], ax=axes0)\n if \"rollDeg\" in dfs_each_collectionName[j].keys():\n axes1.scatter(dfs_each_collectionName[j][\"rollDeg\"],\n [1 for i in range(len(dfs_each_collectionName[j][\"rollDeg\"]))],\n alpha=0.1, s=3, color=colors[j])\n g1.set_title(f\"Phone track of {collectionName}\")\n axes1.set_title(f\"rollDegs of {collectionName}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-05-23T07:09:32.088389Z","iopub.execute_input":"2021-05-23T07:09:32.089011Z","iopub.status.idle":"2021-05-23T07:10:26.154109Z","shell.execute_reply.started":"2021-05-23T07:09:32.088965Z","shell.execute_reply":"2021-05-23T07:10:26.152989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Which collection contains which phone \n\nI checked which phones were included in each collection. ","metadata":{}},{"cell_type":"code","source":"train_phones = glob.glob(\"../input/google-smartphone-decimeter-challenge/train/*/*\")\ntest_phones = glob.glob(\"../input/google-smartphone-decimeter-challenge/test/*/*\")\n\ntrain_data = [item.split(\"/\")[-2:] for item in train_phones]\ntest_data = [item.split(\"/\")[-2:] for item in test_phones]\n\nphoneNames = [item.split(\"/\")[-1:] for item in train_phones] + [item.split(\"/\")[-1:] for item in test_phones]\nphoneNames = set(list(itertools.chain.from_iterable(phoneNames)))\ncollectionNames = [item.split(\"/\")[-2] for item in train_phones] + [item.split(\"/\")[-2] for item in test_phones]\n\ndef add_lack_data(data, phoneNames, collectionNames):\n result = []\n for collectionName in collectionNames:\n for phoneName in phoneNames:\n if [collectionName, phoneName] in data:\n result.append([collectionName, phoneName, 1])\n else:\n result.append([collectionName, phoneName, 0])\n return result\n\ntrain_data = add_lack_data(train_data, phoneNames, collectionNames)\ntest_data = add_lack_data(test_data, phoneNames, collectionNames)\n\ntrain_data = pd.DataFrame(train_data, columns=[\"collectionNames\", \"phoneNames\", \"count\"])\ntest_data = pd.DataFrame(test_data, columns=[\"collectionNames\", \"phoneNames\", \"count\"])\n\nfor phoneName in phoneNames:\n fig, axes = plt.subplots(1, 2, figsize=(20,15))\n plt.subplots_adjust(wspace=0.3, hspace=0.6)\n g1 = seaborn.barplot(data=train_data[train_data[\"phoneNames\"] == phoneName],\n x=\"count\", y=\"collectionNames\", ax=axes[0])\n g1.set_title(f\"{phoneName} in train data\")\n g2 = seaborn.barplot(data=test_data[test_data[\"phoneNames\"] == phoneName],\n x=\"count\", y=\"collectionNames\", ax=axes[1])\n g2.set_title(f\"{phoneName} in test data\")","metadata":{"execution":{"iopub.status.busy":"2021-05-23T08:37:26.629274Z","iopub.execute_input":"2021-05-23T08:37:26.629677Z","iopub.status.idle":"2021-05-23T08:37:57.089700Z","shell.execute_reply.started":"2021-05-23T08:37:26.629645Z","shell.execute_reply":"2021-05-23T08:37:57.088496Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}