{"cells":[{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# For dependencies\n!pip install -U lyft_dataset_sdk  # To load the lidar data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Same as all the other notebooks, symlink the directories to work with LyftDataset API\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's get start with the data from the a random sample. According to the [official doc](https://github.com/lyft/nuscenes-devkit#dataset-structure), a `sample` is a key frame from a 25-45 seconds drive of the car. Each `sample` is accompanied with lidar data from [three different lidar sensors](https://level5.lyft.com/dataset/#data-collection), camera data from seven different cameras, and annotations of 3D objects. In this notebook, we will try to get an interactive visualization of the lidar data with [open3d](https://github.com/intel-isl/Open3D).\n\nTo be more specific, we will visualize only the top lidar in this notbook. Left and right lidars can be visualized in the same way."},{"metadata":{"trusted":true},"cell_type":"code","source":"from lyft_dataset_sdk.lyftdataset import LyftDataset\n\ndata_set = LyftDataset(data_path=\".\", json_path=\"data\")\n\n# We use the first 'sample' as an example ;D\nsample = data_set.sample[0]\n\n# Get the meta data for the lidar data\nlidar_top = data_set.get('sample_data', sample[\"data\"][\"LIDAR_TOP\"])\nlidar_top","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To be able to visualize the lidar point cloud, we will first load the points from disk and prepare them as the `PointCloud` data type of `open3d`."},{"metadata":{"trusted":true},"cell_type":"code","source":"from pathlib import Path\nimport numpy as np\nimport open3d as o3d\nfrom lyft_dataset_sdk.utils.data_classes import LidarPointCloud\n\n# Load the lidar point clouds \nlidar_file = data_set.get('sample_data', sample[\"data\"][\"LIDAR_TOP\"])[\"filename\"]\nlidar_pc = LidarPointCloud.from_file(Path(\"\") / lidar_file)\n\n# Prepare as open3d.geometry.PointCloud \nlidar_np = lidar_pc.points.transpose((1, 0))  # transpose from (3, n) to (n, 3)\nlidar_xyz = lidar_np[:, :3]\nlidar_intensity = lidar_np[:, 3]\n\npcd = o3d.geometry.PointCloud()\npcd.points = o3d.utility.Vector3dVector(lidar_xyz)\npcd","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are 62937 points in this point cloud. Now it's time to start visualize 'em. Note that You will need to fork and run it yourself to show and interact with the plot!"},{"metadata":{"trusted":true},"cell_type":"code","source":"from open3d import JVisualizer\n\nvisualizer = JVisualizer()\nvisualizer.add_geometry(pcd)\nvisualizer.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems showing nothing. Don't be worried. There's a [known issue](http://www.open3d.org/docs/release/tutorial/Basic/jupyter.html#jupyter-visualization) of the fixed initial camera pose. It is not a good camera pose to visualize this point cloud. We can simply change the camera pose with the following controls:\n> - Mouse wheel: zoom in/out\n> - Left mouse button drag: rotate axis\n> - Right mouse button drag: panning\n\nFor this example, you can scroll down the mouse wheel to zoom out and the point cloud will start to show up."},{"metadata":{},"cell_type":"markdown","source":"But this is NOT all of it. The visualization of Lidar points are also available from the official tool. What makes this visualization interesting is that we can customize it for special use cases, for example, visualizing instance segmentation labels!\n\nFirst of all, instance segmentation labels are not provided in the training data. But given 3D Lidar points & bounding boxes, it's not difficult to extract instance segmentation labels given the fact (or assumption) that Lidar points are sparse in 3D space and bounding boxes do not suffer high overlapping with each other.\n\nIn the following code blocks, we implement a vanilla instance segmentation labels extraction function `to_ego_motion_inst_segm` and use it to extract segmentation labels per point and instance labels per point and save them into files."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\nfrom pyquaternion import Quaternion\nfrom lyft_dataset_sdk.utils.data_classes import LidarPointCloud\nfrom lyft_dataset_sdk.utils.geometry_utils import view_points, points_in_box, quaternion_yaw\n\n\nTYPE2CLASS = {\n    'car':0,\n    'pedestrian':1,\n    'animal':2,\n    'other_vehicle':3, \n    'bus':4, \n    'motorcycle':5, \n    'truck':6, \n    'emergency_vehicle':7, \n    'bicycle':8\n} \n\ndef to_ego_motion_inst_segm(sample, data_set, data_path):\n    lidar_data = data_set.get('sample_data', sample[\"data\"][\"LIDAR_TOP\"])\n    lidar_pc = LidarPointCloud.from_file(data_path / lidar_data[\"filename\"])\n\n    ego_pose = data_set.get(\"ego_pose\", lidar_data[\"ego_pose_token\"])\n    cs_pose = data_set.get(\"calibrated_sensor\", lidar_data[\"calibrated_sensor_token\"])\n    points = to_flat_vehicle_coordinates(lidar_pc, ego_pose, cs_pose)\n    num_points = points.shape[1]\n\n    lidar_xyz = points.transpose((1, 0))[:, :3]\n\n    anns = sample[\"anns\"]\n    instance_ids = np.zeros(shape=(num_points), dtype=np.uint32)  # 0: unannotated\n    label_ids = np.ones(shape=(num_points), dtype=np.uint16) * 9  # 9: unannotated\n    boxes = []\n    for idx, ann in enumerate(anns):\n        box = data_set.get_box(ann)\n        yaw = Quaternion(ego_pose[\"rotation\"]).yaw_pitch_roll[0]\n        box.translate(-np.array(ego_pose[\"translation\"]))\n        box.rotate(Quaternion(scalar=np.cos(yaw / 2), vector=[0, 0, np.sin(yaw / 2)]).inverse)\n        \n        xyzlwhyawcls = np.hstack([box.center, box.wlh[[1,0,2]], (box.orientation.yaw_pitch_roll[0], TYPE2CLASS[box.name])])\n        boxes.append(xyzlwhyawcls)\n\n        # if a point belongs more than one box, assigned labels to the last one\n        mask = points_in_box(box, points[:3, :])\n        instance_ids[mask] = idx + 1\n        label_ids[mask] = TYPE2CLASS[box.name]\n    \n    return lidar_xyz, np.asarray(boxes), label_ids, instance_ids\n\ndef to_flat_vehicle_coordinates(pc, pose_record, cs_record):\n    \"\"\"Credit goes to \n     - https://www.kaggle.com/rishabhiitbhu/eda-understanding-the-dataset-with-3d-plots/\n     - https://github.com/lyft/nuscenes-devkit/\n    \"\"\"\n    vehicle_from_sensor = np.eye(4)\n    vehicle_from_sensor[:3, :3] = Quaternion(cs_record[\"rotation\"]).rotation_matrix\n    vehicle_from_sensor[:3, 3] = cs_record[\"translation\"]\n    \n    ego_yaw = Quaternion(pose_record[\"rotation\"]).yaw_pitch_roll[0]\n    rot_vehicle_flat_from_vehicle = np.dot(\n        Quaternion(scalar=np.cos(ego_yaw / 2), vector=[0, 0, np.sin(ego_yaw / 2)]).rotation_matrix,\n        Quaternion(pose_record[\"rotation\"]).inverse.rotation_matrix,\n    )\n    vehicle_flat_from_vehicle = np.eye(4)\n    vehicle_flat_from_vehicle[:3, :3] = rot_vehicle_flat_from_vehicle\n    points = view_points(\n        pc.points[:3, :], np.dot(vehicle_flat_from_vehicle, vehicle_from_sensor), normalize=False\n    )\n    return points\n\nto_ego_motion_inst_segm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pathlib import Path\n\n\nDATA_PATH = Path(\".\")\noutput_path = DATA_PATH / \"trainval\"\noutput_path.mkdir(exist_ok=True, parents=True)\nlidar_xyz, boxes, label_ids, instance_ids = to_ego_motion_inst_segm(sample, data_set, DATA_PATH)\n\nvert_file = output_path / f\"{sample['token']}_vert.npy\"\nnp.save(vert_file, lidar_xyz)\nprint(f\"Writing {vert_file} finished.\")\n\nbbox_file = output_path / f\"{sample['token']}_bbox.npy\"\nnp.save(bbox_file, boxes)\nprint(f\"Writing {bbox_file} finished.\")\n\nsem_label_file = output_path / f\"{sample['token']}_sem_label.npy\"\nnp.save(sem_label_file, label_ids)\nprint(f\"Writing {sem_label_file} finished.\")\n\nins_label_file = output_path / f\"{sample['token']}_ins_label.npy\"\nnp.save(ins_label_file, instance_ids)\nprint(f\"Writing {ins_label_file} finished.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, the segmentation labels/instance labels are loaded as colors for the Lidar point cloud. The colored point cloud can be visualized with open3d's JVisualizer in the same way as before."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport open3d as o3d\n\nfrom open3d import JVisualizer\nimport matplotlib.colors as mcolors\n\ncmap = [mcolors.to_rgb(value) for value in mcolors.TABLEAU_COLORS.values()]\n\nsample_token = sample[\"token\"]\nvert_np = np.load(output_path / f\"{sample_token}_vert.npy\")\nseg_np = np.load(output_path / f\"{sample_token}_sem_label.npy\")\nins_np = np.load(output_path / f\"{sample_token}_ins_label.npy\")\n\n\npcd = o3d.geometry.PointCloud()\npcd.points = o3d.utility.Vector3dVector(vert_np)\npcd.colors = o3d.utility.Vector3dVector(np.asarray([cmap[int(cls)] for cls in seg_np]))  # To visualize segmentation label\n# pcd.colors = o3d.utility.Vector3dVector(np.asarray([cmap[int(ins % len(cmap))] for ins in ins_np]))  # To visualize instance label\n\nvisualizer = JVisualizer()\n\nvisualizer.add_geometry(pcd)\nvisualizer.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's it. Enjoy navigating the world of Lidar data!"},{"metadata":{},"cell_type":"markdown","source":"## Trouble shooting\n\n* The tested version of open3d is `open3d-python==0.7.0.0`. The newest version `open3d==0.8.0.0` is not working because of [this bug](https://github.com/intel-isl/Open3D/issues/949#issuecomment-531886936).\n* The `open3d-python` package needs to be install before the notebook start so that the jupyter widgets can be loaded. This can be done by \"installing custom package\" to the kernel from the settings panel."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}