{"cells":[{"metadata":{"colab_type":"text","id":"6Glr-cdMIdQU"},"cell_type":"markdown","source":"# <span style=\"color:blue\"> Lyft 3D Object Detection for Autonomous Vehicles </span>\n\n<br>\n\n<img src=\"https://s3-prod.crainsnewyork.com/s3fs-public/MAIN-Lyft%20pink%20cars_Buck%20Ennis_i_i.jpg\" height=\"500\" width=\"500\"> \n \n**Self-driving technology** presents a rare opportunity to improve the quality of life in many of our communities. Avoidable collisions, single-occupant commuters, and vehicle emissions are choking cities, while infrastructure strains under rapid urban growth. Autonomous vehicles are expected to redefine transportation and unlock a myriad of societal, environmental, and economic benefits. You can apply your data analysis skills in this competition to advance the state of self-driving technology.\n\n![](https://storage.googleapis.com/kaggle-media/competitions/Lyft-Kaggle/Kaggle-01.png)\n\n\n**This dataset** aims to democratize access to such data, and foster innovation in higher-level autonomy functions for everyone, everywhere. By conducting a competition, we hope to encourage the research community to focus on hard problems in this space—namely, 3D object detection over semantic maps.\n\nIn **this competition**, you will build and optimize algorithms based on a large-scale dataset. This dataset features the raw sensor camera inputs as perceived by a fleet of multiple, high-end, autonomous vehicles in a restricted geographic area. \n\n\n##  References\n\n- [Lyft: Quick EDA and creating useful files](https://www.kaggle.com/xhlulu/lyft-quick-eda-and-creating-useful-files) by @xhlulu\n- **[Official Devkit for the public 2019 Lyft Level 5 AV Dataset](https://github.com/lyft/nuscenes-devkit) by @iglovikov**"},{"metadata":{},"cell_type":"markdown","source":"# Data\n\n\nYou will need the **LIDAR, image**, map and data files for both train and test (```test_images.zip```, ```test_lidar.zip```, etc.). You may also need the train.csv, which includes the sample annotations in the form expected for submissions. The ```sample_submission.csv``` file contains all of the sample ```Ids``` for the test set.\n\nThe data files (```test_data.zip```, ```train_data.zip```) are in JSON format.\n\n<br>\n\n- **train_data.zip** and **test_data.zip** - contains JSON files with multiple tables. The most important is ```sample_data.json```, which contains the primary identifiers used in the competition, as well as links to key image / lidar information.\n    \n- **train_images.zip** and **test_images.zip** - contains .jpeg files corresponding to samples in ```sample_data.json```\n- **train_lidar.zip** and **test_lidar.zip** - contains .jpeg files corresponding to samples in ```sample_data.json```\n- **train_maps.zip** and **test_maps.zip** - contains maps of the entire sample area.\n- **train.csv** - contains all ```sample_tokens``` in the train set, as well as annotations in the required format for all train set objects.\n- **sample_submission.csv** - contains all ```sample_tokens``` in the test set, with empty predictions.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!ls ../input/3d-object-detection-for-autonomous-vehicles","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"BCUppFwZ_5y6"},"cell_type":"markdown","source":"# Lyft Level 5 AV dataset and nuScenes devkit tutorial\n\n### <span style=\"color:red\"> IMPORTANT </span>\n\n> This is a modification of the official devkit tutorial: https://github.com/lyft/nuscenes-devkit . I modified the code for running it here! and I'll add new stuff. Check the official repository for more impormation.\n\n\nWelcome to the Level 5 AV dataset & nuScenes SDK tutorial!\n\nThis notebook is based on the original nuScenes tutorial notebook (https://www.nuscenes.org/) and was adjusted for the Level 5 AV dataset."},{"metadata":{"colab_type":"text","id":"UC5ekf8G_5y9"},"cell_type":"markdown","source":"## Introduction to the dataset structure\n\nIn this part of the tutorial, let us go through a top-down introduction of our database. Our dataset comprises of elemental building blocks that are the following:\n\n1. `scene` - 25-45 seconds snippet of a car's journey.\n2. `sample` - An annotated snapshot of a scene at a particular timestamp.\n3. `sample_data` - Data collected from a particular sensor.\n4. `sample_annotation` - An annotated instance of an object within our interest.\n5. `instance` - Enumeration of all object instance we observed.\n6. `category` - Taxonomy of object categories (e.g. vehicle, human). \n7. `attribute` - Property of an instance that can change while the category remains the same.\n8. `visibility` - (currently not used)\n9. `sensor` - A specific sensor type.\n10. `calibrated sensor` - Definition of a particular sensor as calibrated on a particular vehicle.\n11. `ego_pose` - Ego vehicle poses at a particular timestamp.\n12. `log` - Log information from which the data was extracted.\n13. `map` - Map data that is stored as binary semantic masks from a top-down view."},{"metadata":{},"cell_type":"markdown","source":"Let's get started! Make sure that you have a local copy of a dataset (for download instructions, see https://level5.lyft.com/dataset/). Then, adjust `dataroot` below to point to your local dataset path. If everything is set up correctly, you should be able to execute the following cell successfully."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install lyft-dataset-sdk","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the SDK\n%matplotlib inline\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks @rishabhiitbhu for this comment: https://www.kaggle.com/seshurajup/lyft-level-5-av-dataset-notebook-from-github#625566"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Load the dataset\n# Adjust the dataroot parameter below to point to your local dataset path.\n# The correct dataset path contains at least the following four folders (or similar): images, lidar, maps, v1.0.1-train\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data = LyftDataset(data_path='.', json_path='/kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data', verbose=True)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"5sdZ0TzI_5y9"},"cell_type":"markdown","source":"### 1. Scene"},{"metadata":{"colab_type":"text","id":"QJiMCRqQ_5y-"},"cell_type":"markdown","source":"Let's take a look at the scenes that we have in the loaded database. This example dataset only has one scene, but there are many more to come."},{"metadata":{"colab":{},"colab_type":"code","id":"7kWoijOW_5y-","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"level5data.list_scenes()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"a6028rl__5zA"},"cell_type":"markdown","source":"Let's look at a scene's **metadata**"},{"metadata":{"colab":{},"colab_type":"code","id":"-LbG8nXC_5zB","trusted":true},"cell_type":"code","source":"my_scene = level5data.scene[0]\nmy_scene","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"yya09DjQ_5zD"},"cell_type":"markdown","source":"### 2. Sample"},{"metadata":{"colab_type":"text","id":"LMEw6eWj_5zE"},"cell_type":"markdown","source":"We define `sample` as an ***annotated keyframe of a scene at a given timestamp***. A keyframe is a frame where the time-stamps of data from all the sensors should be very close to the time-stamp of the sample it points to.\n\nNow, let us look at the first annotated sample in this scene."},{"metadata":{"colab":{},"colab_type":"code","id":"k-7caTTR_5zE","scrolled":true,"trusted":true},"cell_type":"code","source":"my_sample_token = my_scene[\"first_sample_token\"]\n# my_sample_token = level5data.get(\"sample\", my_sample_token)[\"next\"]  # proceed to next sample\n\nlevel5data.render_sample(my_sample_token)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"g5AF88zq_5zG"},"cell_type":"markdown","source":"Let's examine its **metadata** (click `output`)"},{"metadata":{"colab":{},"colab_type":"code","id":"Fa44_jBk_5zH","scrolled":false,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"my_sample = level5data.get('sample', my_sample_token)\nmy_sample","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"rCzHU8-4_5zK"},"cell_type":"markdown","source":"A useful method is  `list_sample()` which lists all related `sample_data` keyframes and `sample_annotation` associated with a `sample` which we will discuss in detail in the subsequent parts."},{"metadata":{"colab":{},"colab_type":"code","id":"Kz8dCNQ7_5zK","scrolled":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"level5data.list_sample(my_sample['token'])","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"1qaXI4A0zw2e"},"cell_type":"markdown","source":"Instead of looking at camera and lidar data separately, we can also project the lidar pointcloud into camera images:"},{"metadata":{"colab":{},"colab_type":"code","id":"Ps3fmoMmzqHq","trusted":true},"cell_type":"code","source":"level5data.render_pointcloud_in_image(sample_token = my_sample[\"token\"],\n                                      dot_size = 1,\n                                      camera_channel = 'CAM_FRONT')","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"As0fsJJU_5zN"},"cell_type":"markdown","source":"### 3. Sample_data"},{"metadata":{"colab_type":"text","id":"SUyx-JsG_5zO"},"cell_type":"markdown","source":"The dataset contains data that is collected from a full sensor suite. Hence, for each snapshot of a scene, we provide references to a family of data that is collected from these sensors. \n\nWe provide a `data` key to access these:"},{"metadata":{"colab":{},"colab_type":"code","id":"u8UfbGsG_5zO","trusted":true},"cell_type":"code","source":"my_sample['data']","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"91iSBRY3_5zS"},"cell_type":"markdown","source":"Notice that the keys are referring to the different sensors that form our sensor suite. Let's take a look at the metadata of a `sample_data` taken from `CAM_FRONT`."},{"metadata":{"colab":{},"colab_type":"code","id":"EMQWn0JC_5zS","trusted":true},"cell_type":"code","source":"sensor_channel = 'CAM_FRONT'  # also try this e.g. with 'LIDAR_TOP'\nmy_sample_data = level5data.get('sample_data', my_sample['data'][sensor_channel])\nmy_sample_data","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"FTEiZfAQ_5zU"},"cell_type":"markdown","source":"We can also render the `sample_data` at a particular sensor. "},{"metadata":{"colab":{},"colab_type":"code","id":"2hRXFQcF_5zV","trusted":true},"cell_type":"code","source":"level5data.render_sample_data(my_sample_data['token'])","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"-sEaYTs0_5zY"},"cell_type":"markdown","source":"### 4. Sample_annotation"},{"metadata":{"colab_type":"text","id":"gJH-Bi7j_5zY"},"cell_type":"markdown","source":"`sample_annotation` refers to any ***bounding box defining the position of an object seen in a sample***. All location data is given with respect to the global coordinate system. Let's examine an example from our `sample` above."},{"metadata":{"colab":{},"colab_type":"code","id":"K0Cfkl-b_5zZ","trusted":true},"cell_type":"code","source":"my_annotation_token = my_sample['anns'][16]\nmy_annotation =  my_sample_data.get('sample_annotation', my_annotation_token)\nmy_annotation","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"Ob11cIsQ_5zc"},"cell_type":"markdown","source":"We can also render an annotation to have a closer look."},{"metadata":{"colab":{},"colab_type":"code","id":"AOoScoo-_5zd","trusted":true},"cell_type":"code","source":"level5data.render_annotation(my_annotation_token)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"XPdUyPCa_5zf"},"cell_type":"markdown","source":"### 5. Instance"},{"metadata":{"colab_type":"text","id":"VNQRtjB__5zf"},"cell_type":"markdown","source":"Object instance are instances that need to be detected or tracked by an AV (e.g a particular vehicle, pedestrian). Let us examine an instance metadata"},{"metadata":{"colab":{},"colab_type":"code","id":"KwWWgzIp_5zg","trusted":true},"cell_type":"code","source":"my_instance = level5data.instance[100]\nmy_instance","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"4fuDLND3_5zh"},"cell_type":"markdown","source":"We generally track an instance across different frames in a particular scene. However, we do not track them across different scenes. In this example, we have 16 annotated samples for this instance across a particular scene."},{"metadata":{"colab":{},"colab_type":"code","id":"RdlcLLuS_5zh","trusted":true},"cell_type":"code","source":"instance_token = my_instance['token']\nlevel5data.render_instance(instance_token)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"bn7sIX7s_5zk"},"cell_type":"markdown","source":"An instance record takes note of its first and last annotation token. Let's render them"},{"metadata":{"colab":{},"colab_type":"code","id":"BsNbgW6j_5zl","trusted":true},"cell_type":"code","source":"print(\"First annotated sample of this instance:\")\nlevel5data.render_annotation(my_instance['first_annotation_token'])","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"z0I8tugr_5zm","trusted":true},"cell_type":"code","source":"print(\"Last annotated sample of this instance\")\nlevel5data.render_annotation(my_instance['last_annotation_token'])","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"NlOoRIg1_5zo"},"cell_type":"markdown","source":"### 6. Category"},{"metadata":{"colab_type":"text","id":"641qDwoq_5zu"},"cell_type":"markdown","source":"A `category` is the object assignment of an annotation.  Let's look at the category table we have in our database. The table contains the taxonomy of different object categories and also list the subcategories (delineated by a period). "},{"metadata":{"colab":{},"colab_type":"code","id":"nTfY_wU6_5zu","trusted":true},"cell_type":"code","source":"level5data.list_categories()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"K7yRmSRT_5zv"},"cell_type":"markdown","source":"A category record contains the name and the description of that particular category."},{"metadata":{"colab":{},"colab_type":"code","id":"mHjq-umn_5zw","trusted":true},"cell_type":"code","source":"level5data.category[2]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"XRv2lRe0_5zx"},"cell_type":"markdown","source":"### 7. Attribute"},{"metadata":{"colab_type":"text","id":"ttRuoaBK_5zy"},"cell_type":"markdown","source":"An `attribute` is a property of an instance that may change throughout different parts of a scene while the category remains the same. Here we list the provided attributes and the number of annotations associated with a particular attribute."},{"metadata":{"colab":{},"colab_type":"code","id":"x86EPZCZ_5zy","trusted":true},"cell_type":"code","source":"level5data.list_attributes()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"p27KXuP1_5z0"},"cell_type":"markdown","source":"Let's take a look at an example how an attribute may change over one scene"},{"metadata":{"colab":{},"colab_type":"code","id":"lPEphmyC_5z1","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"for my_instance in level5data.instance:\n    first_token = my_instance['first_annotation_token']\n    last_token = my_instance['last_annotation_token']\n    nbr_samples = my_instance['nbr_annotations']\n    current_token = first_token\n\n    i = 0\n    found_change = False\n    while current_token != last_token:\n        current_ann = level5data.get('sample_annotation', current_token)\n        current_attr = level5data.get('attribute', current_ann['attribute_tokens'][0])['name']\n\n        if i == 0:\n            pass\n        elif current_attr != last_attr:\n            print(\"Changed from `{}` to `{}` at timestamp {} out of {} annotated timestamps\".format(last_attr, current_attr, i, nbr_samples))\n            found_change = True\n\n        next_token = current_ann['next']\n        current_token = next_token\n        last_attr = current_attr\n        i += 1","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"sLR0RzAa_5z8"},"cell_type":"markdown","source":"### 8. Sensor"},{"metadata":{"colab_type":"text","id":"3LU4ypwj_5z9"},"cell_type":"markdown","source":"The Level 5 dataset consists of data collected from our full sensor suite which consists of:\n- 1 x LIDAR, (up to three in final dataset)\n- 7 x cameras, "},{"metadata":{"colab":{},"colab_type":"code","id":"4YtlV8fw_5z-","scrolled":true,"trusted":true},"cell_type":"code","source":"level5data.sensor","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"ADs0zLkw_50A"},"cell_type":"markdown","source":"Every `sample_data` has a record on which `sensor` the data is collected from (note the \"channel\" key)"},{"metadata":{"colab":{},"colab_type":"code","id":"p_-K5d8D_50B","trusted":true},"cell_type":"code","source":"level5data.sample_data[10]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"EEeD7_Uk_50C"},"cell_type":"markdown","source":"### 9. Calibrated_sensor"},{"metadata":{"colab_type":"text","id":"G57Y-TAb_50D"},"cell_type":"markdown","source":"`calibrated_sensor` consists of the definition of a particular sensor (lidar/camera) as calibrated on a particular vehicle. Let us look at an example."},{"metadata":{"colab":{},"colab_type":"code","id":"2WIyooA3_50D","scrolled":true,"trusted":true},"cell_type":"code","source":"level5data.calibrated_sensor[0]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"_bMaRkvp_50F"},"cell_type":"markdown","source":"Note that the `translation` and the `rotation` parameters are given with respect to the ego vehicle body frame. "},{"metadata":{"colab_type":"text","id":"Y82HB-bX_50F"},"cell_type":"markdown","source":"### 10. ego_pose"},{"metadata":{"colab_type":"text","id":"NqEzApx4_50G"},"cell_type":"markdown","source":"`ego_pose` contains information about the location (encoded in `translation`) and the orientation (encoded in `rotation`) of the ego vehicle body frame, with respect to the global coordinate system."},{"metadata":{"colab":{},"colab_type":"code","id":"_ECy5VW9_50G","trusted":true},"cell_type":"code","source":"level5data.ego_pose[0]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"tS4_vJS-_50I"},"cell_type":"markdown","source":"### 11. log\n\nThe `log` table contains log information from which the data was extracted. A `log` record corresponds to one journey of our ego vehicle along a predefined route. Let's check the number of logs and the metadata of a log."},{"metadata":{"colab":{},"colab_type":"code","id":"jEEy5Usm_50J","trusted":true},"cell_type":"code","source":"print(\"Number of `logs` in our loaded database: {}\".format(len(level5data.log)))","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"jwlSPvD4_50L","trusted":true},"cell_type":"code","source":"level5data.log[0]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"tjajeuU4_50O"},"cell_type":"markdown","source":"Notice that it contains a variety of information such as the date and location of the log collected. It also gives out information about the map from where the data was collected. Note that one log can contain multiple non-overlapping scenes."},{"metadata":{"colab_type":"text","id":"7zCRo8u7_50O"},"cell_type":"markdown","source":"### 12. Map"},{"metadata":{"colab_type":"text","id":"HHx9DRis_50P"},"cell_type":"markdown","source":"Map information is currently stored in a 2D rasterized image. Let's check the number of maps and metadata of a map."},{"metadata":{"colab":{},"colab_type":"code","id":"3dWIvv0D_50Q","trusted":true},"cell_type":"code","source":"print(\"There are {} maps masks in the loaded dataset\".format(len(level5data.map)))","execution_count":null,"outputs":[]},{"metadata":{"colab":{},"colab_type":"code","id":"T8xyK3wb_50R","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"level5data.map[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Memory!!\n\nThe map can e.g. be displayed in the background of top-down views:\n> I don't run this, we need more RAM..."},{"metadata":{"trusted":true},"cell_type":"code","source":"sensor_channel = 'LIDAR_TOP'\n#my_sample_data = level5data.get('sample_data', my_sample['data'][sensor_channel])\n# The following call can be slow and requires a lot of memory\n#level5data.render_sample_data(my_sample_data['token'], underlay_map = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset and Devkit Basics"},{"metadata":{},"cell_type":"markdown","source":"Let's get a bit **technical.**\n\nThe NuScenes class holds several tables. Each table is a list of records, and each record is a dictionary. For example the first record of the category table is stored at:"},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.category[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The category table is simple: it holds the fields `name` and `description`. It also has a `token` field, which is a unique record identifier. Since the record is a dictionary, the token can be accessed like so:"},{"metadata":{"trusted":true},"cell_type":"code","source":"cat_token = level5data.category[0]['token']\ncat_token","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you know the `token` for any record in the DB you can retrieve the record by doing"},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.get('category', cat_token)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"_As you can notice, we have recovered the same record!_"},{"metadata":{},"cell_type":"markdown","source":"OK, that was easy. Let's try something harder. Let's look at the `sample_annotation` table."},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.sample_annotation[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This also has a `token` field (they all do). In addition, it has several fields of the format [a-z]*\\_token, _e.g._ instance_token. These are foreign keys in database speak, meaning they point to another table. \nUsing `level5data.get()` we can grab any of these in constant time.\n\nNote that in our dataset, we don't provide `num_lidar_pts` and set it to `-1` to indicate this."},{"metadata":{"trusted":true},"cell_type":"code","source":"one_instance = level5data.get('instance', level5data.sample_annotation[0]['instance_token'])\none_instance","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This points to the `instance` table. This table enumerate the object _instances_ we have encountered in each \nscene. This way we can connect all annotations of a particular object.\n\nIf you look carefully at the tables, you will see that the sample_annotation table points to the instance table, \nbut the instance table doesn't list all annotations that point to it. \n\nSo how can we recover all sample_annotations for a particular object instance? There are two ways:\n\n1. `Use level5data.field2token()`. Let's try it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ann_tokens = level5data.field2token('sample_annotation', 'instance_token', one_instance['token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This returns a list of all sample_annotation records with the `'instance_token'` == `one_instance['token']`. Let's store these in a set for now"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"ann_tokens_field2token = set(ann_tokens)\n\nann_tokens_field2token","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `level5data.field2token()` method is generic and can be used in any similar situation.\n\n2. For certain situation, we provide some reverse indices in the tables themselves. This is one such example. "},{"metadata":{},"cell_type":"markdown","source":"The instance record has a field `first_annotation_token` which points to the first annotation in time of this instance. \nRecovering this record is easy."},{"metadata":{"trusted":true},"cell_type":"code","source":"ann_record = level5data.get('sample_annotation', one_instance['first_annotation_token'])\nann_record","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can traverse all annotations of this instance using the \"next\" field. Let's try it. "},{"metadata":{"trusted":true},"cell_type":"code","source":"ann_tokens_traverse = set()\nann_tokens_traverse.add(ann_record['token'])\nwhile not ann_record['next'] == \"\":\n    ann_record = level5data.get('sample_annotation', ann_record['next'])\n    ann_tokens_traverse.add(ann_record['token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let's assert that we recovered the same ann_records as we did using level5data.field2token:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(ann_tokens_traverse == ann_tokens_field2token)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reverse indexing and short-cuts\n\nThe dataset tables are normalized, meaning that each piece of information is only given once.\nFor example, there is one `map` record for each `log` record. Looking at the schema you will notice that the `map` table has a `log_token` field, but that the `log` table does not have a corresponding `map_token` field. But there are plenty of situations where you have a `log`, and want to find the corresponding `map`! So what to do? You can always use the `level5data.field2token()` method, but that is slow and inconvenient. The devkit therefore adds reverse mappings for some common situations including this one.\n\nFurther, there are situations where one needs to go through several tables to get a certain piece of information. \nConsider, for example, the category name of a `sample_annotation`. The `sample_annotation` table doesn't hold this information since the category is an instance level constant. Instead the `sample_annotation` table points to a record in the `instance` table. This, in turn, points to a record in the `category` table, where finally the `name` fields stores the required information.\n\nSince it is quite common to want to know the category name of an annotation, we add a `category_name` field to the `sample_annotation` table during initialization of the NuScenes class.\n\nIn this section, we list the short-cuts and reverse indices that are added to the `NuScenes` class during initialization. These are all created in the `NuScenes.__make_reverse_index__()` method."},{"metadata":{},"cell_type":"markdown","source":"### Reverse indices\nThe devkit adds two reverse indices by default.\n* A `map_token` field is added to the `log` records.\n* The `sample` records have shortcuts to all `sample_annotations` for that record as well as `sample_data` key-frames. Confer `level5data.list_sample()` method in the previous section for more details on this."},{"metadata":{},"cell_type":"markdown","source":"### Shortcuts"},{"metadata":{},"cell_type":"markdown","source":"The sample_annotation table has a \"category_name\" shortcut."},{"metadata":{},"cell_type":"markdown","source":"_Using shortcut:_"},{"metadata":{"trusted":true},"cell_type":"code","source":"catname = level5data.sample_annotation[0]['category_name']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"_Not using shortcut:_"},{"metadata":{"trusted":true},"cell_type":"code","source":"ann_rec = level5data.sample_annotation[0]\ninst_rec = level5data.get('instance', ann_rec['instance_token'])\ncat_rec = level5data.get('category', inst_rec['category_token'])\n\nprint(catname == cat_rec['name'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The sample_data table has \"channel\" and \"sensor_modality\" shortcuts:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Shortcut\nchannel = level5data.sample_data[0]['channel']\n\n# No shortcut\nsd_rec = level5data.sample_data[0]\ncs_record = level5data.get('calibrated_sensor', sd_rec['calibrated_sensor_token'])\nsensor_record = level5data.get('sensor', cs_record['sensor_token'])\n\nprint(channel == sensor_record['channel'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Visualizations\n\nWe provide list and rendering methods. These are meant both as convenience methods during development, and as tutorials for building your own visualization methods. They are implemented in the NuScenesExplorer class, with shortcuts through the NuScenes class itself."},{"metadata":{},"cell_type":"markdown","source":"### List methods\nThere are three list methods available."},{"metadata":{},"cell_type":"markdown","source":"1. `list_categories()` lists all categories, counts and statistics of width/length/height in meters and aspect ratio."},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.list_categories()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. `list_attributes()` lists all attributes and counts."},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.list_attributes()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. `list_scenes()` lists all scenes in the loaded DB."},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"level5data.list_scenes()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Render"},{"metadata":{},"cell_type":"markdown","source":"First, let's plot a lidar point cloud in an image. Lidar allows us to accurately map the surroundings in 3D."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_sample = level5data.sample[10]\nlevel5data.render_pointcloud_in_image(my_sample['token'], pointsensor_channel='LIDAR_TOP')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also plot all annotations across all sample data for that sample."},{"metadata":{"trusted":true},"cell_type":"code","source":"my_sample = level5data.sample[20]\n\n# The rendering command below is commented out because it tends to crash in notebooks\n# level5data.render_sample(my_sample['token'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Or if we only want to render a particular sensor, we can specify that."},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.render_sample_data(my_sample['data']['CAM_FRONT'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Additionally we can aggregate the point clouds from multiple sweeps to get a denser point cloud."},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.render_sample_data(my_sample['data']['LIDAR_TOP'], nsweeps=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can even render a specific annotation."},{"metadata":{"trusted":true},"cell_type":"code","source":"level5data.render_annotation(my_sample['anns'][22])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we can render a full scene as a video. There are two options here:\n1. level5data.render_scene_channel() renders the video for a particular channel. (HIT ESC to exit)\n2. level5data.render_scene() renders the video for all surround view camera channels.\n\n**NOTE: These methods use OpenCV for rendering, which doesn't always play nice with IPython Notebooks. If you experience any issues please run these lines from the command line. **"},{"metadata":{"trusted":true},"cell_type":"code","source":"#my_scene_token = level5data.scene[0][\"token\"]\n#level5data.render_scene_channel(my_scene_token, 'CAM_FRONT')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is also a method level5data.render_scene() which renders the video for all camera channels."},{"metadata":{"trusted":true},"cell_type":"code","source":"#level5data.render_scene(my_scene_token)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, let us visualize all scenes on the map for a particular location."},{"metadata":{"trusted":true},"cell_type":"code","source":"#level5data.render_egoposes_on_map(log_location='Palo Alto')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Continue\n\nI'll keep adding stuff here, mainly EDA and Visualization. If you want to learn more about this, please check the post\n[Some important links to get started](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/108613#latest-625428)\n\n\n- https://medium.com/@SmartLabAI/3d-object-detection-from-lidar-data-with-deep-learning-95f6d400399a\n- https://github.com/timzhang642/3D-Machine-Learning\n- https://towardsdatascience.com/the-state-of-3d-object-detection-f65a385f67a8\n\n\nAlso you can read about resources and experiences here [[new quota] Competition Expectations + experiences](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/discussion/108609#latest-625543)"},{"metadata":{"colab":{},"colab_type":"code","id":"7USmHJFYeVU_"},"cell_type":"raw","source":"# put your code here"}],"metadata":{"colab":{"collapsed_sections":[],"name":"Level 5 AV Dataset Tutorial","provenance":[],"version":"0.3.2"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"}},"nbformat":4,"nbformat_minor":1}