{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Let's understand the dataset\n\n### Given files\n\n- **train_data.zip** and **test_data.zip** - contains JSON files with multiple tables. The most important is ```sample_data.json```, which contains the primary identifiers used in the competition, as well as links to key image / lidar information.\n    \n- **train_images.zip** and **test_images.zip** - contains .jpeg files corresponding to samples in ```sample_data.json```\n- **train_lidar.zip** and **test_lidar.zip** - contains .bin files corresponding to samples in ```sample_data.json```\n- **train_maps.zip** and **test_maps.zip** - contains maps of the entire sample area.\n- **train.csv** - contains all ```sample_tokens``` in the train set, as well as annotations in the required format for all train set objects.\n- **sample_submission.csv** - contains all ```sample_tokens``` in the test set, with empty predictions.\n"},{"metadata":{},"cell_type":"markdown","source":"**Updates**\n\n* V34: 3D interactive visualization of a sample added, uses latest lyft sdk\n* V33: Added info about `nuscenes_viz` (refer to the end of this kernel)\n* V30: Improved 3D plots\n* V26: Added boxes in the 3D plot"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install -U git+https://github.com/lyft/nuscenes-devkit moviepy >> /dev/tmp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nfrom IPython.display import HTML","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pdb\nimport cv2\nimport pandas as pd\nimport numpy as np\nfrom PIL import Image\nfrom pathlib import Path\nfrom matplotlib import pyplot as plt\nfrom mpl_toolkits.mplot3d import axes3d, Axes3D\n\n# Load the SDK\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset, LyftDatasetExplorer, Quaternion, view_points\nfrom lyft_dataset_sdk.utils.data_classes import LidarPointCloud\n\nfrom moviepy.editor import ImageSequenceClip\nfrom tqdm import tqdm_notebook as tqdm\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Introduction to the dataset structure\n\nLet us go through a top-down introduction of the database. The dataset comprises of elemental building blocks that are the following:\n\n1. `scene` - 25-45 seconds snippet of a car's journey.\n2. `sample` - An annotated snapshot of a scene at a particular timestamp.\n3. `sample_data` - Data collected from a particular sensor.\n4. `sample_annotation` - An annotated instance of an object within our interest.\n5. `instance` - Enumeration of all object instance we observed.\n6. `category` - Taxonomy of object categories (e.g. vehicle, human). \n7. `attribute` - Property of an instance that can change while the category remains the same.\n8. `visibility` - (currently not used)\n9. `sensor` - A specific sensor type.\n10. `calibrated sensor` - Definition of a particular sensor as calibrated on a particular vehicle.\n11. `ego_pose` - Ego vehicle poses at a particular timestamp.\n12. `log` - Log information from which the data was extracted.\n13. `map` - Map data that is stored as binary semantic masks from a top-down view."},{"metadata":{},"cell_type":"markdown","source":"The data comes in the form of many interlocking tables and formats. The JSON files all contain single tables with identifying tokens that can be used to join with other files / tables. The images and lidar files all correspond to a sample in `sample_data.json`\n\nWe will be using [lyft's Dataset SDK](https://github.com/lyft/nuscenes-devkit/) for doing data analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"# gotta do this for LyftDataset SDK, it expects folders to be named as `images`, `maps`, `lidar`\n\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata = LyftDataset(data_path='.', json_path='data/', verbose=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The LyftDataset class holds several tables. Each table is a list of records, and each record is a dictionary. For example the first record of the category table is stored at:"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.category[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The category table is simple: it holds the fields `name` and `description`. It also has a `token` field, which is a unique record identifier. Since the record is a dictionary, the token can be accessed like so:"},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_token = lyftdata.category[0]['token']\ncat_token","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you know the `token` for any record in the DB you can retrieve the record by doing"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.get('category', cat_token)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"_As you can notice, we have recovered the same record!_"},{"metadata":{},"cell_type":"markdown","source":"Ok. Let's try something harder. Let's look at the `sample_annotation` table."},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.sample_annotation[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This also has a `token` field (they all do). In addition, it has several fields of the format [a-z]*\\_token, _e.g._ instance_token. These are foreign keys in database terminology, meaning they point to another table. \nUsing `lyftdata.get()` we can grab any of these in constant time. For example, let's look at the first attribute record."},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.get('attribute', lyftdata.sample_annotation[0]['attribute_tokens'][0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The attribute record indicates about what was the state of the concerned object when it was annotated."},{"metadata":{},"cell_type":"markdown","source":"Let's get started with the database schema, one by one starting with the `Scenes`"},{"metadata":{},"cell_type":"markdown","source":"# Scenes\n\nA scene is a 25-45s long sequence of consecutive frames extracted from a log. A frame (also called a `sample`) is a collection of sensor outputs (images, lidar points) at a given timestamp\n\n```\nscene {\n   \"token\":                   <str> -- Unique record identifier.\n   \"name\":                    <str> -- Short string identifier.\n   \"description\":             <str> -- Longer description of the scene.\n   \"log_token\":               <str> -- Foreign key. Points to log from where the data was extracted.\n   \"nbr_samples\":             <int> -- Number of samples in this scene.\n   \"first_sample_token\":      <str> -- Foreign key. Points to the first sample in scene.\n   \"last_sample_token\":       <str> -- Foreign key. Points to the last sample in scene.\n}\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.scene[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each scene provides the first sample token and the last sample token, we can see there are 126 sample records (`nbr_samples`) in between these two."},{"metadata":{},"cell_type":"markdown","source":"### Let's take a look at train.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/3d-object-detection-for-autonomous-vehicles/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"train dataframe's `Id` column contains tokens (unique identifiers) of train sample records present in `sample` table and `PredictionString` contains corresponding ground truth annotations (bounding boxes) for different object categories"},{"metadata":{"trusted":true},"cell_type":"code","source":"token0 = train.iloc[0]['Id']\ntoken0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll be using `token0` to as our reference sample token"},{"metadata":{},"cell_type":"markdown","source":"# Sample"},{"metadata":{},"cell_type":"markdown","source":"A `sample` is defined as an annotated keyframe of a `scene` at a given timestamp. A sample is data collected at (approximately) the same timestamp as part of a single LIDAR sweep. \n\n```\nsample {\n   \"token\":                   <str> -- Unique record identifier.\n   \"timestamp\":               <int> -- Unix time stamp.\n   \"scene_token\":             <str> -- Foreign key pointing to the scene.\n   \"next\":                    <str> -- Foreign key. Sample that follows this in time. Empty if end of scene.\n   \"prev\":                    <str> -- Foreign key. Sample that precedes this in time. Empty if start of scene.\n}\n```"},{"metadata":{},"cell_type":"markdown","source":"Remember, `token0` is a token to a particular sample record in `sample` data table (`sample.json`), let's look at that sample using lyft SDK's inbuilt `.get` function"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_sample = lyftdata.get('sample', token0)\nmy_sample","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3D interactive visualization of a sample"},{"metadata":{},"cell_type":"markdown","source":"We can visualize a sample interactively using lyft SDK's inbuilt `render_sample_3d_interactive` functionality"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.render_sample_3d_interactive(my_sample['token'], render_sample=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see what else have we got .."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_sample.keys()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So our `token0` points to `my_sample` record in `sample` data table. This sample record has got 7 camera images and 3 LIDAR data files, which can be fetched using their respective tokens, from where? none other than `sample_data` table."},{"metadata":{},"cell_type":"markdown","source":"# Sample data"},{"metadata":{},"cell_type":"markdown","source":"`sample_data` is a sensor data e.g. image or point cloud.\n\n```\nsample_data {\n   \"token\":                   <str> -- Unique record identifier.\n   \"sample_token\":            <str> -- Foreign key. Sample to which this sample_data is associated.\n   \"ego_pose_token\":          <str> -- Foreign key.\n   \"calibrated_sensor_token\": <str> -- Foreign key.\n   \"filename\":                <str> -- Relative path to data-blob on disk.\n   \"fileformat\":              <str> -- Data file format.\n   \"width\":                   <int> -- If the sample data is an image, this is the image width in pixels.\n   \"height\":                  <int> -- If the sample data is an image, this is the image height in pixels.\n   \"timestamp\":               <int> -- Unix time stamp.\n   \"is_key_frame\":            <bool> -- True if sample_data is part of key_frame, else False.\n   \"next\":                    <str> -- Foreign key. Sample data from the same sensor that follows this in time. Empty if end of scene.\n   \"prev\":                    <str> -- Foreign key. Sample data from the same sensor that precedes this in time. Empty if start of scene.\n}\n```"},{"metadata":{},"cell_type":"markdown","source":"The dataset contains data that is collected from a full sensor suite. Hence, for each snapshot of a scene, we are provided with references to a family of data that is collected from these sensors. "},{"metadata":{},"cell_type":"markdown","source":"The full sensor suite consists of:\n* 7 cameras,\n* 3 LIDARs"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.sensor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can use `data` key of a given sample to access one of these, like:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sensor = 'CAM_FRONT'\ncam_front = lyftdata.get('sample_data', my_sample['data'][sensor])\ncam_front","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Every `sample_data` has a record of the `sensor` from which the data was collected from (note the \"channel\" key). Let's see image taken by front camera (CAM_FRONT)"},{"metadata":{"trusted":true},"cell_type":"code","source":"img = Image.open(cam_front['filename'])\nimg","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also use SDK's inbuilt `render_sample_data` to visualize `cam_front` data (remember `cam_front` is a record in `sample_data` table)"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.render_sample_data(cam_front['token'], with_anns=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"let's take a look at the LIDAR data associated with `my_sample`"},{"metadata":{"trusted":true},"cell_type":"code","source":"lidar_top = lyftdata.get('sample_data', my_sample['data']['LIDAR_TOP']) # selecting LIDAR_TOP out of all LIDARs\nlidar_top","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can use `LidarPointCloud` to read LIDAR data files"},{"metadata":{"trusted":true},"cell_type":"code","source":"pc = LidarPointCloud.from_file(Path(lidar_top['filename']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pc.points.shape # x, y, z, intensity","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize these lidar points"},{"metadata":{"trusted":true},"cell_type":"code","source":"axes_limits = [\n    [-30, 50], # X axis range\n    [-30, 20], # Y axis range\n    [-3, 10]   # Z axis range\n]\naxes_str = ['X', 'Y', 'Z']\n\ndef display_frame_statistics(lidar_points, points=0.2):\n    \"\"\"\n    Displays statistics for a single frame. Draws 3D plot of the lidar point cloud data and point cloud\n    projections to various planes.\n    \n    Parameters\n    ----------\n    lidar_points: lidar data points \n    points          : Fraction of lidar points to use. Defaults to `0.2`, e.g. 20%.\n    \"\"\"\n    \n    points_step = int(1. / points)\n    point_size = 0.01 * (1. / points)\n    pc_range = range(0, lidar_points.shape[1], points_step)\n    pc_frame = lidar_points[:, pc_range]\n    def draw_point_cloud(ax, title, axes=[0, 1, 2]):\n        \"\"\"Convenient method for drawing various point cloud projections as a part of frame statistics\"\"\"\n        ax.set_facecolor('black')\n        ax.grid(False)\n        ax.scatter(*pc_frame[axes, :], s=point_size, c='white', cmap='grey')\n        if len(axes) == 3: # 3D configs\n            text_color = 'white'\n            ax.set_xlim3d([-10, 30])\n            ax.set_ylim3d(*axes_limits[axes[1]])\n            ax.set_zlim3d(*axes_limits[axes[2]])\n            ax.set_zlabel('{} axis'.format(axes_str[axes[2]]), color='white')\n        else: # 2D configs\n            text_color = 'black' # the `figure` is white\n            ax.set_xlim(*axes_limits[axes[0]])\n            ax.set_ylim(*axes_limits[axes[1]])\n        ax.set_title(title, color=text_color)\n        ax.set_xlabel('{} axis'.format(axes_str[axes[0]]), color=text_color)\n        ax.set_ylabel('{} axis'.format(axes_str[axes[1]]), color=text_color)\n            \n    # Draw point cloud data as 3D plot\n    f2 = plt.figure(figsize=(15, 8))\n    ax2 = f2.add_subplot(111, projection='3d')\n    # make the panes transparent\n    ax2.xaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax2.yaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax2.zaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    draw_point_cloud(ax2, '3D plot')\n    \n    plt.show()\n    plt.close(f2)\n    # Draw point cloud data as plane projections\n    f, ax3 = plt.subplots(3, 1, figsize=(15, 25))\n#     f.set_facecolor('black')\n    draw_point_cloud(\n        ax3[0], \n        'XZ projection (Y = 0)', #, the car is moving in direction left to right', ?\n        axes=[0, 2] # X and Z axes\n    )\n    draw_point_cloud(\n        ax3[1], \n        'XY projection (Z = 0)', #, the car is moving in direction left to right',? \n        axes=[0, 1] # X and Y axes\n    )\n    draw_point_cloud(\n        ax3[2], \n        'YZ projection (X = 0)', #, the car is moving towards the graph plane', ?\n        axes=[1, 2] # Y and Z axes\n    )\n    plt.show()\n    plt.close(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_frame_statistics(pc.points, points=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also use SDK's inbuilt `render_sample_data` to visualize `lidar_top` (remember `lidar_top` is a record in `sample_data` table, just like `cam_front` is )"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.render_sample_data(lidar_top['token'], with_anns=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, above plots show a particular LIDAR sensor (`TOP_LIDAR`), Each sensor is mounted on a ego vehicle aka Autonomous vehicle (AV), here comes `ego_pose`"},{"metadata":{},"cell_type":"markdown","source":"## ego_pose"},{"metadata":{},"cell_type":"markdown","source":"`ego_pose` contains information about the location (encoded in `translation`) and the orientation (encoded in `rotation`) of the ego vehicle body frame, with respect to the global coordinate system."},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.get('ego_pose', cam_front['ego_pose_token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ego_pose parameters are used to bring points from global coordinates system (say bounding boxes' corners) to ego vehicle's frame of reference. As we know that each ego vehicle has a bunch of sensors, each sensor has got some calibration parameters w.r.t ego vehicle"},{"metadata":{},"cell_type":"markdown","source":"## caliberated_sensor"},{"metadata":{},"cell_type":"markdown","source":"`calibrated_sensor` consists of the definition of a particular sensor (lidar/radar/camera) as calibrated on a particular vehicle. Let us look at an example."},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.get('calibrated_sensor', cam_front['calibrated_sensor_token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All extrinsic parameters are given with respect to the ego vehicle body frame. We use caliberate_sensor's parameters to bring points from ego vehicle's frame of reference to sensor's frame of reference. It is used to plot bounding boxes in the images and lidar point cloud."},{"metadata":{},"cell_type":"markdown","source":"We'll need sensor calibration and ego_pose parameters when plotting bounding boxes together with the lidar point cloud. We'll see this later on, before that let's visualize other stuff which `my_sample` has got, starting with bounding box annotations"},{"metadata":{},"cell_type":"markdown","source":"# Annotations"},{"metadata":{},"cell_type":"markdown","source":"`sample_annotation` refers to a bounding box defining the position of an object seen in a sample.\nAll location data is given with respect to the global coordinate system.\n```\nsample_annotation {\n   \"token\":                   <str> -- Unique record identifier.\n   \"sample_token\":            <str> -- Foreign key. NOTE: this points to a sample NOT a sample_data since annotations are done on the sample level taking all relevant sample_data into account.\n   \"instance_token\":          <str> -- Foreign key. Which object instance is this annotating. An instance can have multiple annotations over time.\n   \"attribute_tokens\":        <str> [n] -- Foreign keys. List of attributes for this annotation. Attributes can change over time, so they belong here, not in the object table.\n   \"visibility_token\":        <str> -- Foreign key. Visibility may also change over time. If no visibility is annotated, the token is an empty string.\n   \"translation\":             <float> [3] -- Bounding box location in meters as center_x, center_y, center_z.\n   \"size\":                    <float> [3] -- Bounding box size in meters as width, length, height.\n   \"rotation\":                <float> [4] -- Bounding box orientation as quaternion: w, x, y, z.\n   \"num_lidar_pts\":           <int> -- Number of lidar points in this box. Points are counted during the lidar sweep identified with this sample.\n   \"num_radar_pts\":           <int> -- Number of radar points in this box. Points are counted during the radar sweep identified with this sample. This number is summed across all radar sensors without any invalid point filtering.\n   \"next\":                    <str> -- Foreign key. Sample annotation from the same object instance that follows this in time. Empty if this is the last annotation for this object.\n   \"prev\":                    <str> -- Foreign key. Sample annotation from the same object instance that precedes this in time. Empty if this is the first annotation for this object.\n}\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_annotation = lyftdata.get('sample_annotation', my_sample['anns'][0])\nmy_annotation","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each annotation has all location data with respect to global coordinate system. We will have to take care of rotation, translation and size into account before finalizing the final bounding box coordinates on the sample images luckily we have an inbuilt function for that"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_box = lyftdata.get_box(my_annotation['token'])\nmy_box # Box class instance","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_box.center, my_box.wlh # center coordinates + width, length and height","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see our `my_annotation` (remember it is a single object annotation out of the many present in `my_sample`) using the SDK's inbuilt `render_annotation` function"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.render_annotation(my_annotation['token'], margin=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each annotation has an `attribute`:"},{"metadata":{},"cell_type":"markdown","source":"# Attributes"},{"metadata":{},"cell_type":"markdown","source":"An attribute is a property of an instance that can change while the category remains the same.\n Example: a vehicle being parked/stopped/moving, and whether or not a bicycle has a rider.\n```\nattribute {\n   \"token\":                   <str> -- Unique record identifier.\n   \"name\":                    <str> -- Attribute name.\n   \"description\":             <str> -- Attribute description.\n}\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_attribute1 = lyftdata.get('attribute', my_annotation['attribute_tokens'][0])\nmy_attribute2 = lyftdata.get('attribute', my_annotation['attribute_tokens'][1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_attribute1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"my_attribute2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like our subject is standstill"},{"metadata":{},"cell_type":"markdown","source":"# Instances"},{"metadata":{},"cell_type":"markdown","source":"Object instance are instances that need to be detected or tracked by an autonomous vehicle (e.g a particular vehicle, pedestrian). Let us examine an instance metadata"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_instance = lyftdata.get('instance', my_annotation['instance_token'])\nmy_instance","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.render_instance(my_instance['token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"People generally track an instance across different frames in a particular scene. That's why we have `first_annotation_token` and `last_annotation_token` in `my_instance`"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"First annotated sample of this instance:\")\nlyftdata.render_annotation(my_instance['first_annotation_token'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Last annotated sample of this instance\")\nlyftdata.render_annotation(my_instance['last_annotation_token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So far we have seen that each sample data point has a bunch of images taken from different cameras, a bunch of lidar data points taken from different lidar sensors, it has annotations for the objects in those images, attributes associated with those annotations. Let's visualize everything which this data point has got in itself using SDK's inbuit `render_sample` function"},{"metadata":{"trusted":true},"cell_type":"code","source":"lyftdata.render_sample(token0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3D visualization of a scene"},{"metadata":{"trusted":true},"cell_type":"code","source":"my_scene = lyftdata.get('scene',  my_sample['scene_token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize `LIDAR_TOP` of all the samples of our `my_scene` in 3D. Here's how we are gonna do this:\n* Each lidar point cloud is in [lidar ranger](https://en.wikipedia.org/wiki/Laser_rangefinder)'s frame of reference, we have to get each point in vehicle's frame of reference. \n* Each bounding box corners are in global coordinate system, we get them to ego vehicles frame of reference using ego_pose parameters. \n* Once we have the lidar point and box corners in vehicle's frame of reference, we plot the points using matplotlib's scatter function and draw the box edges using matplotlib's plot function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_lidar_points(lidar_token):\n    '''Get lidar point cloud in the frame of the ego vehicle'''\n    sd_record = lyftdata.get(\"sample_data\", lidar_token)\n    sensor_modality = sd_record[\"sensor_modality\"]\n    \n    # Get aggregated point cloud in lidar frame.\n    sample_rec = lyftdata.get(\"sample\", sd_record[\"sample_token\"])\n    chan = sd_record[\"channel\"]\n    ref_chan = \"LIDAR_TOP\"\n    pc, times = LidarPointCloud.from_file_multisweep(\n        lyftdata, sample_rec, chan, ref_chan, num_sweeps=1\n    )\n    # Compute transformation matrices for lidar point cloud\n    cs_record = lyftdata.get(\"calibrated_sensor\", sd_record[\"calibrated_sensor_token\"])\n    pose_record = lyftdata.get(\"ego_pose\", sd_record[\"ego_pose_token\"])\n    vehicle_from_sensor = np.eye(4)\n    vehicle_from_sensor[:3, :3] = Quaternion(cs_record[\"rotation\"]).rotation_matrix\n    vehicle_from_sensor[:3, 3] = cs_record[\"translation\"]\n    \n    ego_yaw = Quaternion(pose_record[\"rotation\"]).yaw_pitch_roll[0]\n    rot_vehicle_flat_from_vehicle = np.dot(\n        Quaternion(scalar=np.cos(ego_yaw / 2), vector=[0, 0, np.sin(ego_yaw / 2)]).rotation_matrix,\n        Quaternion(pose_record[\"rotation\"]).inverse.rotation_matrix,\n    )\n    vehicle_flat_from_vehicle = np.eye(4)\n    vehicle_flat_from_vehicle[:3, :3] = rot_vehicle_flat_from_vehicle\n    points = view_points(\n        pc.points[:3, :], np.dot(vehicle_flat_from_vehicle, vehicle_from_sensor), normalize=False\n    )\n    return points","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_box(box, axis, view, colors, normalize=False, linewidth=1.0):\n    '''Plot boxes in the 3d figure'''\n    corners = view_points(box.corners(), view, normalize=normalize)#\n    def draw_rect(selected_corners, color):\n        prev = selected_corners[-1]\n        for corner in selected_corners:\n            axis.plot([prev[0], corner[0]], [prev[1], corner[1]], [prev[2], corner[2]], color=color, linewidth=linewidth)\n            prev = corner\n\n    # Draw the sides\n    for i in range(4):\n        axis.plot(\n            [corners.T[i][0], corners.T[i + 4][0]],\n            [corners.T[i][1], corners.T[i + 4][1]],\n            [corners.T[i][2], corners.T[i + 4][2]],\n            color=colors[2],\n            linewidth=linewidth,\n        )\n\n    # Draw front (first 4 corners) and rear (last 4 corners) rectangles(3d)/lines(2d)\n    draw_rect(corners.T[:4], colors[0]) #4x3\n    draw_rect(corners.T[4:], colors[1])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_3d_plot(idx, lidar_token):\n    '''Plot the lidar + annotations on a 3D figure'''\n    # sample lidar point cloud\n    lidar_points = get_lidar_points(lidar_token)\n    points = 0.5 # fraction of lidar_points to plot, to reduce the clutter\n    points_step = int(1. / points)\n    pc_range = range(0, lidar_points.shape[1], points_step)\n    lidar_points = lidar_points[:, pc_range]\n    \n    # Get boxes, instead of current sensor's coordinate frame, use vehicle frame which is aligned to z-plane in world\n    _, boxes, _ = lyftdata.get_sample_data(\n        lidar_token, flat_vehicle_coordinates=True\n    )\n    fig = plt.figure(figsize=(15, 8))\n    ax = fig.add_subplot(111, projection='3d')                    \n    point_size = 0.01 * (1. / points) # size of the dots on plot\n    ax.set_facecolor('black')\n    ax.grid(False)\n    ax.scatter(*lidar_points, s=point_size, c='white', cmap='gray')\n    for box in boxes:\n        c = np.array(lyftdata.explorer.get_color(box.name)) / 255.0\n        plot_box(box, ax, view=np.eye(3), colors=(c, c, c), linewidth=0.5)\n    ax.set_xlim3d(-40, 40)\n    ax.set_ylim3d(-40, 40)\n    ax.set_zlim3d(-4, 40)\n    \n    # make the panes transparent\n    ax.xaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax.yaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax.zaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax.set_title(lidar_token, color='white')\n    filename = 'tmp/frame_{0:0>4}.png'.format(idx)\n    plt.savefig(filename)\n    plt.close(fig)\n    return filename","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir tmp # a temporary folder to contain plot jpegs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's take a quick look at the 3d Plot\nfirst_sample_token = my_scene['first_sample_token']\nsample = lyftdata.get('sample', first_sample_token)\nlidar_token = sample['data']['LIDAR_TOP']\nfilename = draw_3d_plot(0, lidar_token)\nImage.open(filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize `my_scene` in the form of a video"},{"metadata":{"trusted":true},"cell_type":"code","source":"frames = []\nfirst_sample_token = my_scene['first_sample_token']\ntoken = first_sample_token\nfor i in tqdm(range(my_scene['nbr_samples'])):\n    sample = lyftdata.get('sample', token)\n    lidar_token = sample['data']['LIDAR_TOP']\n    filename = draw_3d_plot(i, lidar_token)\n    frames += [filename]\n    token = sample['next']\n#     break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"clip = ImageSequenceClip(frames, fps=5)\nclip.write_gif('pcl_data.gif', fps=5);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nfrom IPython import display","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('pcl_data.gif','rb') as f:\n    display.Image(data=f.read(), format='png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -r tmp/*","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also use lyft SDK's `render_scene` function to render a scene like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# The rendering command below is commented out because it tends to crash in notebooks,\n# lyftdata.render_scene(my_scene['token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" I've uploaded it on YouTube\n \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML('<iframe width=\"700\" height=\"500\" src=\"https://www.youtube.com/embed/Vs8H8Fs-zTs\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3D visualization tool: `nuscenes_viz`"},{"metadata":{},"cell_type":"markdown","source":"I've built this awesome tool to interactively visualize lidar point cloud and corresponding bounding boxes in 3D using [mayavi](https://github.com/enthought/mayavi). It's called `nuscenes_viz`\n\nCheckout the source code: [here](https://github.com/pyaf/nuscenes_viz)\n\nHere's a demo video:"},{"metadata":{"trusted":true},"cell_type":"code","source":"HTML('<iframe width=\"700\" height=\"430\" src=\"https://www.youtube.com/embed/ivmiN4zvRTo\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Do upvote if you liked this kernel :)"},{"metadata":{},"cell_type":"markdown","source":"## References\n\n* https://github.com/lyft/nuscenes-devkit/\n* https://github.com/nutonomy/nuscenes-devkit/blob/master/schema.md\n* https://github.com/enginBozkurt/Visualizing-lidar-data/"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}