{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Các tệp đã cho\n\n- **train_data.zip** và **test_data.zip** - chứa các tệp JSON có nhiều bảng. Điều quan trọng nhất là ```sample_data.json```, chứa các giá trị nhận dạng chính được sử dụng trong cuộc thi, cũng như các liên kết đến thông tin hình ảnh/lidar chính.\n    \n- **train_images.zip** và **test_images.zip** - chứa các tệp .jpeg tương ứng với các mẫu trong ```sample_data.json```\n- **train_lidar.zip** và **test_lidar.zip** - chứa các tệp .bin tương ứng với các mẫu trong ```sample_data.json```\n- **train_maps.zip** và **test_maps.zip** - chứa các bản đồ của toàn bộ khu vực mẫu.\n- **train.csv** - chứa tất cả ```sample_tokens``` trong nhóm tàu, cũng như các chú thích ở định dạng bắt buộc cho tất cả các đối tượng của nhóm tàu.\n- **sample_submission.csv** - chứa tất cả ```sample_tokens``` trong bộ thử nghiệm, với các dự đoán trống.","metadata":{}},{"cell_type":"code","source":"!pip install -U git+https://github.com/lyft/nuscenes-devkit moviepy >> /dev/tmp","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-06-05T07:49:54.558938Z","iopub.execute_input":"2023-06-05T07:49:54.559285Z","iopub.status.idle":"2023-06-05T07:50:36.397513Z","shell.execute_reply.started":"2023-06-05T07:49:54.559235Z","shell.execute_reply":"2023-06-05T07:50:36.396088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"import module \"warnings\" để xử lý cảnh báo trong Python. Module này cho phép chúng ta quản lý và kiểm soát các cảnh báo được sinh ra bởi các thư viện và code ","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nfrom IPython.display import HTML","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:50:36.400156Z","iopub.execute_input":"2023-06-05T07:50:36.400573Z","iopub.status.idle":"2023-06-05T07:50:36.407615Z","shell.execute_reply.started":"2023-06-05T07:50:36.400515Z","shell.execute_reply":"2023-06-05T07:50:36.406497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pdb\nimport cv2\nimport pandas as pd\nimport numpy as np\nfrom PIL import Image\nfrom pathlib import Path\nfrom matplotlib import pyplot as plt\nfrom mpl_toolkits.mplot3d import axes3d, Axes3D\n\n# Load the SDK\nfrom lyft_dataset_sdk.lyftdataset import LyftDataset, LyftDatasetExplorer, Quaternion, view_points\nfrom lyft_dataset_sdk.utils.data_classes import LidarPointCloud\n\nfrom moviepy.editor import ImageSequenceClip\nfrom tqdm import tqdm_notebook as tqdm\n\n%matplotlib inline","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2023-06-05T07:50:36.409486Z","iopub.execute_input":"2023-06-05T07:50:36.409897Z","iopub.status.idle":"2023-06-05T07:50:40.043780Z","shell.execute_reply.started":"2023-06-05T07:50:36.409830Z","shell.execute_reply":"2023-06-05T07:50:40.042481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Giới thiệu về cấu trúc dữ liệu\nHãy cùng đi qua một giới thiệu từ trên xuống về cơ sở dữ liệu. Tập dữ liệu bao gồm các khối xây dựng cơ bản sau đây:\n\n\n* scene - Đoạn video dài 25-45 giây về hành trình của một chiếc ô tô.\n* sample - Một hình ảnh được chú thích của một scene tại một thời điểm cụ thể.\n* sample_data - Dữ liệu được thu thập từ một cảm biến cụ thể.\n* sample_annotation - Một phiên bản được chú thích của một đối tượng trong tầm quan tâm của chúng ta.\n* instance - Liệt kê tất cả các phiên bản đối tượng mà chúng ta đã quan sát được.\n* category - Phân loại các loại đối tượng (ví dụ: phương tiện, con người).\n* attribute - Thuộc tính của một phiên bản có thể thay đổi trong khi loại vẫn giữ nguyên.\n* visibility - (hiện không được sử dụng)\n* sensor - Một loại cảm biến cụ thể.\n* calibrated sensor - Định nghĩa của một cảm biến cụ thể đã được hiệu chuẩn trên một phương tiện cụ thể.\n* ego_pose - Vị trí của xe chủ động vào một thời điểm cụ thể.\n* log - Thông tin nhật ký từ đó dữ liệu được trích xuất.\n* map - Dữ liệu bản đồ được lưu trữ dưới dạng mặt nạ ngữ nghĩa nhị phân từ góc nhìn từ trên xuống.","metadata":{}},{"cell_type":"markdown","source":"Dữ liệu có dạng nhiều bảng và định dạng lồng vào nhau. Tất cả các tệp JSON đều chứa các bảng đơn có mã thông báo xác định có thể được sử dụng để nối với các tệp/bảng khác. Tất cả các hình ảnh và tệp lidar đều tương ứng với một mẫu trong sample_data.json\n\nChúng tôi sẽ sử dụng SDK Bộ dữ liệu của lyft để thực hiện phân tích dữ liệu\n\n[lyft's Dataset SDK](https://github.com/lyft/nuscenes-devkit/) ","metadata":{}},{"cell_type":"code","source":"# LyftDataset SDK, nó yêu cầu các thư mục được đặt tên là `images`, `maps`, `lidar`\n\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_images images\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_maps maps\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_lidar lidar\n!ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data data","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:50:40.045548Z","iopub.execute_input":"2023-06-05T07:50:40.045903Z","iopub.status.idle":"2023-06-05T07:50:44.417593Z","shell.execute_reply.started":"2023-06-05T07:50:40.045840Z","shell.execute_reply":"2023-06-05T07:50:44.416291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Cụ thể, các đối số được truyền vào:\n* \n* data_path='.': Đây là đường dẫn tới thư mục chứa dữ liệu Lyft. Trong trường hợp này, '.' đại diện cho thư mục hiện tại, nghĩa là thư mục trong đó đoạn mã này đang được thực thi.\n* json_path='data/': Đây là đường dẫn tới thư mục chứa các tệp JSON của dữ liệu Lyft. Thư mục này được liên kết tới trong dòng mã trước đó bằng lệnh !ln -s /kaggle/input/3d-object-detection-for-autonomous-vehicles/train_data data.\n* verbose=True: Đối số này xác định rằng bạn muốn hiển thị thông tin chi tiết khi tạo đối tượng LyftDataset.\n* Sau khi đối tượng LyftDataset được khởi tạo, bạn có thể sử dụng nó để truy cập và làm việc với các bảng dữ liệu và chức năng trong SDK của Lyft, như category, sample_annotation, scene, vv.","metadata":{}},{"cell_type":"code","source":"lyftdata = LyftDataset(data_path='.', json_path='data/', verbose=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:50:44.422366Z","iopub.execute_input":"2023-06-05T07:50:44.423056Z","iopub.status.idle":"2023-06-05T07:51:14.435093Z","shell.execute_reply.started":"2023-06-05T07:50:44.422981Z","shell.execute_reply":"2023-06-05T07:51:14.434018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thông tin về số lượng bản ghi trong các bảng dữ liệu Lyft được tải và các thời gian tải dữ liệu.\n* 9 category: Số lượng danh mục (loại đối tượng) có sẵn trong dữ liệu Lyft.\n* 18 attribute: Số lượng thuộc tính (đặc điểm) có sẵn trong dữ liệu Lyft.\n* 4 visibility: Số lượng trạng thái hiển thị (có thể nhìn thấy hoặc không nhìn thấy) có sẵn trong dữ liệu Lyft.\n* 18421 instance: Số lượng trường hợp (ví dụ: xe cụ thể) có sẵn trong dữ liệu Lyft.\n* 10 sensor: Số lượng cảm biến có sẵn trong dữ liệu Lyft.\n* 148 calibrated_sensor: Số lượng cảm biến được hiệu chuẩn có sẵn trong dữ liệu Lyft.\n* 177789 ego_pose: Số lượng vị trí ego (vị trí của xe tự hành) có sẵn trong dữ liệu Lyft.\n* 180 log: Số lượng nhật ký (dữ liệu ghi lại) có sẵn trong dữ liệu Lyft.\n* 180 scene: Số lượng cảnh (giao lộ hoặc đoạn đường cụ thể) có sẵn trong dữ liệu Lyft.\n* 22680 sample: Số lượng mẫu (khung hình cụ thể) có sẵn trong dữ liệu Lyft.\n* 189504 sample_data: Số lượng dữ liệu mẫu (hình ảnh, lidar, radar, vv.) có sẵn trong dữ liệu Lyft.\n* 638179 sample_annotation: Số lượng chú thích mẫu (chú thích vị trí, hình dạng, vv.) có sẵn trong dữ liệu Lyft.\n* 1 map: Số lượng bản đồ có sẵn trong dữ liệu Lyft.","metadata":{}},{"cell_type":"markdown","source":"LyftDataset chứa một số bảng. Mỗi bảng là một danh sách các bản ghi và mỗi bản ghi là một từ điển. Ví dụ: bản ghi đầu tiên của bảng danh mục được lưu trữ tại:","metadata":{}},{"cell_type":"markdown","source":"lyftdata.category[0]: Đây là ví dụ về cách truy cập vào bản ghi đầu tiên trong bảng category. Mỗi bản ghi trong bảng này là một từ điển (dictionary) chứa các trường thông tin như name (tên) và description (mô tả) của danh mục. Bằng cách sử dụng chỉ số [0], chúng ta có thể truy cập vào bản ghi đầu tiên.","metadata":{}},{"cell_type":"code","source":"lyftdata.category[0]","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.439614Z","iopub.execute_input":"2023-06-05T07:51:14.439916Z","iopub.status.idle":"2023-06-05T07:51:14.450220Z","shell.execute_reply.started":"2023-06-05T07:51:14.439869Z","shell.execute_reply":"2023-06-05T07:51:14.448864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"cat_token = lyftdata.category[0]['token']: Đoạn mã này trích xuất giá trị của trường token (mã định danh duy nhất) từ bản ghi đầu tiên trong bảng category và lưu trữ vào biến cat_token.","metadata":{}},{"cell_type":"code","source":"cat_token = lyftdata.category[0]['token']\ncat_token","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.452223Z","iopub.execute_input":"2023-06-05T07:51:14.452680Z","iopub.status.idle":"2023-06-05T07:51:14.462068Z","shell.execute_reply.started":"2023-06-05T07:51:14.452607Z","shell.execute_reply":"2023-06-05T07:51:14.461096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"lyftdata.get('category', cat_token): Đoạn mã này sử dụng phương thức get của LyftDataset để lấy lại bản ghi từ bảng category thông qua token. Bằng cách truyền 'category' làm đối số đầu tiên và cat_token làm đối số thứ hai, chúng ta có thể truy xuất lại bản ghi ban đầu từ bảng category. Kết quả là bản ghi ban đầu và bản ghi được lấy lại là giống nhau.","metadata":{}},{"cell_type":"code","source":"lyftdata.get('category', cat_token)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.463705Z","iopub.execute_input":"2023-06-05T07:51:14.464107Z","iopub.status.idle":"2023-06-05T07:51:14.472688Z","shell.execute_reply.started":"2023-06-05T07:51:14.463986Z","shell.execute_reply":"2023-06-05T07:51:14.471693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_As you can notice, we have recovered the same record!_","metadata":{}},{"cell_type":"markdown","source":"lyftdata.sample_annotation[0]: Đây là ví dụ về cách truy cập vào bản ghi đầu tiên trong bảng sample_annotation. Bản ghi này chứa thông tin về chú thích của một mẫu (khung hình cụ thể) trong dữ liệu Lyft.","metadata":{}},{"cell_type":"code","source":"lyftdata.sample_annotation[0]","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.474251Z","iopub.execute_input":"2023-06-05T07:51:14.474595Z","iopub.status.idle":"2023-06-05T07:51:14.484217Z","shell.execute_reply.started":"2023-06-05T07:51:14.474546Z","shell.execute_reply":"2023-06-05T07:51:14.483261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This also has a `token` field (they all do). In addition, it has several fields of the format [a-z]*\\_token, _e.g._ instance_token. These are foreign keys in database terminology, meaning they point to another table. \nUsing `lyftdata.get()` we can grab any of these in constant time. For example, let's look at the first attribute record.","metadata":{}},{"cell_type":"markdown","source":" truy vấn thông tin về thuộc tính của chú thích mẫu đầu tiên trong bảng sample_annotation. Trong bản ghi đó, trường attribute_tokens chứa danh sách mã định danh của các thuộc tính liên quan. Bằng cách truy cập vào phần tử đầu tiên của danh sách attribute_tokens và sử dụng phương thức get của LyftDataset, chúng ta có thể lấy lại thông tin về thuộc tính tương ứng.","metadata":{}},{"cell_type":"code","source":"lyftdata.get('attribute', lyftdata.sample_annotation[0]['attribute_tokens'][0])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.486033Z","iopub.execute_input":"2023-06-05T07:51:14.486379Z","iopub.status.idle":"2023-06-05T07:51:14.495227Z","shell.execute_reply.started":"2023-06-05T07:51:14.486313Z","shell.execute_reply":"2023-06-05T07:51:14.494438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The attribute record indicates about what was the state of the concerned object when it was annotated.","metadata":{}},{"cell_type":"markdown","source":"Let's get started with the database schema, one by one starting with the `Scenes`","metadata":{}},{"cell_type":"markdown","source":"# Scenes\n\nA scene is a 25-45s long sequence of consecutive frames extracted from a log. A frame (also called a `sample`) is a collection of sensor outputs (images, lidar points) at a given timestamp\n\n```\nscene {\n   \"token\":                   <str> -- Unique record identifier.\n   \"name\":                    <str> -- Short string identifier.\n   \"description\":             <str> -- Longer description of the scene.\n   \"log_token\":               <str> -- Foreign key. Points to log from where the data was extracted.\n   \"nbr_samples\":             <int> -- Number of samples in this scene.\n   \"first_sample_token\":      <str> -- Foreign key. Points to the first sample in scene.\n   \"last_sample_token\":       <str> -- Foreign key. Points to the last sample in scene.\n}\n```","metadata":{}},{"cell_type":"markdown","source":"cách truy cập vào bản ghi đầu tiên trong bảng scene. Mỗi bản ghi trong bảng này chứa thông tin về một cảnh (giao lộ hoặc đoạn đường cụ thể) trong dữ liệu Lyft.","metadata":{}},{"cell_type":"code","source":"lyftdata.scene[0]","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.497591Z","iopub.execute_input":"2023-06-05T07:51:14.497986Z","iopub.status.idle":"2023-06-05T07:51:14.506681Z","shell.execute_reply.started":"2023-06-05T07:51:14.497916Z","shell.execute_reply":"2023-06-05T07:51:14.505750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each scene provides the first sample token and the last sample token, we can see there are 126 sample records (`nbr_samples`) in between these two.","metadata":{}},{"cell_type":"markdown","source":"### Let's take a look at train.csv","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/3d-object-detection-for-autonomous-vehicles/train.csv')\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:14.508191Z","iopub.execute_input":"2023-06-05T07:51:14.508581Z","iopub.status.idle":"2023-06-05T07:51:16.197888Z","shell.execute_reply.started":"2023-06-05T07:51:14.508515Z","shell.execute_reply":"2023-06-05T07:51:16.196718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"train dataframe's `Id` column contains tokens (unique identifiers) of train sample records present in `sample` table and `PredictionString` contains corresponding ground truth annotations (bounding boxes) for different object categories","metadata":{}},{"cell_type":"code","source":"token0 = train.iloc[0]['Id']\ntoken0","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:16.199600Z","iopub.execute_input":"2023-06-05T07:51:16.199976Z","iopub.status.idle":"2023-06-05T07:51:16.207189Z","shell.execute_reply.started":"2023-06-05T07:51:16.199913Z","shell.execute_reply":"2023-06-05T07:51:16.206233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll be using `token0` to as our reference sample token","metadata":{}},{"cell_type":"markdown","source":"# Sample","metadata":{}},{"cell_type":"markdown","source":"A `sample` is defined as an annotated keyframe of a `scene` at a given timestamp. A sample is data collected at (approximately) the same timestamp as part of a single LIDAR sweep. \n\n```\nsample {\n   \"token\":                   <str> -- Unique record identifier.\n   \"timestamp\":               <int> -- Unix time stamp.\n   \"scene_token\":             <str> -- Foreign key pointing to the scene.\n   \"next\":                    <str> -- Foreign key. Sample that follows this in time. Empty if end of scene.\n   \"prev\":                    <str> -- Foreign key. Sample that precedes this in time. Empty if start of scene.\n}\n```","metadata":{}},{"cell_type":"markdown","source":"Remember, `token0` is a token to a particular sample record in `sample` data table (`sample.json`), let's look at that sample using lyft SDK's inbuilt `.get` function","metadata":{}},{"cell_type":"code","source":"my_sample = lyftdata.get('sample', token0)\nmy_sample","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:16.208926Z","iopub.execute_input":"2023-06-05T07:51:16.209270Z","iopub.status.idle":"2023-06-05T07:51:16.220958Z","shell.execute_reply.started":"2023-06-05T07:51:16.209200Z","shell.execute_reply":"2023-06-05T07:51:16.219869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3D interactive visualization of a sample","metadata":{}},{"cell_type":"markdown","source":"We can visualize a sample interactively using lyft SDK's inbuilt `render_sample_3d_interactive` functionality","metadata":{}},{"cell_type":"code","source":"lyftdata.render_sample_3d_interactive(my_sample['token'], render_sample=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:16.222760Z","iopub.execute_input":"2023-06-05T07:51:16.223158Z","iopub.status.idle":"2023-06-05T07:51:19.111552Z","shell.execute_reply.started":"2023-06-05T07:51:16.223087Z","shell.execute_reply":"2023-06-05T07:51:19.110672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see what else have we got ..","metadata":{}},{"cell_type":"code","source":"my_sample.keys()","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:19.112763Z","iopub.execute_input":"2023-06-05T07:51:19.113149Z","iopub.status.idle":"2023-06-05T07:51:19.119157Z","shell.execute_reply.started":"2023-06-05T07:51:19.113107Z","shell.execute_reply":"2023-06-05T07:51:19.118425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So our `token0` points to `my_sample` record in `sample` data table. This sample record has got 7 camera images and 3 LIDAR data files, which can be fetched using their respective tokens, from where? none other than `sample_data` table.","metadata":{}},{"cell_type":"markdown","source":"# Sample data","metadata":{}},{"cell_type":"markdown","source":"`sample_data` is a sensor data e.g. image or point cloud.\n\n```\nsample_data {\n   \"token\":                   <str> -- Unique record identifier.\n   \"sample_token\":            <str> -- Foreign key. Sample to which this sample_data is associated.\n   \"ego_pose_token\":          <str> -- Foreign key.\n   \"calibrated_sensor_token\": <str> -- Foreign key.\n   \"filename\":                <str> -- Relative path to data-blob on disk.\n   \"fileformat\":              <str> -- Data file format.\n   \"width\":                   <int> -- If the sample data is an image, this is the image width in pixels.\n   \"height\":                  <int> -- If the sample data is an image, this is the image height in pixels.\n   \"timestamp\":               <int> -- Unix time stamp.\n   \"is_key_frame\":            <bool> -- True if sample_data is part of key_frame, else False.\n   \"next\":                    <str> -- Foreign key. Sample data from the same sensor that follows this in time. Empty if end of scene.\n   \"prev\":                    <str> -- Foreign key. Sample data from the same sensor that precedes this in time. Empty if start of scene.\n}\n```","metadata":{}},{"cell_type":"markdown","source":"The dataset contains data that is collected from a full sensor suite. Hence, for each snapshot of a scene, we are provided with references to a family of data that is collected from these sensors. ","metadata":{}},{"cell_type":"markdown","source":"The full sensor suite consists of:\n* 7 cameras,\n* 3 LIDARs","metadata":{}},{"cell_type":"code","source":"lyftdata.sensor","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:19.120504Z","iopub.execute_input":"2023-06-05T07:51:19.120991Z","iopub.status.idle":"2023-06-05T07:51:19.135081Z","shell.execute_reply.started":"2023-06-05T07:51:19.120946Z","shell.execute_reply":"2023-06-05T07:51:19.133854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can use `data` key of a given sample to access one of these, like:","metadata":{}},{"cell_type":"code","source":"sensor = 'CAM_FRONT'\ncam_front = lyftdata.get('sample_data', my_sample['data'][sensor])\ncam_front","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:19.136798Z","iopub.execute_input":"2023-06-05T07:51:19.137322Z","iopub.status.idle":"2023-06-05T07:51:19.145108Z","shell.execute_reply.started":"2023-06-05T07:51:19.137237Z","shell.execute_reply":"2023-06-05T07:51:19.143892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Every `sample_data` has a record of the `sensor` from which the data was collected from (note the \"channel\" key). Let's see image taken by front camera (CAM_FRONT)","metadata":{}},{"cell_type":"code","source":"img = Image.open(cam_front['filename'])\nimg","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:19.146686Z","iopub.execute_input":"2023-06-05T07:51:19.147139Z","iopub.status.idle":"2023-06-05T07:51:19.635246Z","shell.execute_reply.started":"2023-06-05T07:51:19.147094Z","shell.execute_reply":"2023-06-05T07:51:19.633889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also use SDK's inbuilt `render_sample_data` to visualize `cam_front` data (remember `cam_front` is a record in `sample_data` table)","metadata":{}},{"cell_type":"code","source":"lyftdata.render_sample_data(cam_front['token'], with_anns=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:19.637716Z","iopub.execute_input":"2023-06-05T07:51:19.638393Z","iopub.status.idle":"2023-06-05T07:51:20.106924Z","shell.execute_reply.started":"2023-06-05T07:51:19.638099Z","shell.execute_reply":"2023-06-05T07:51:20.105770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"let's take a look at the LIDAR data associated with `my_sample`","metadata":{}},{"cell_type":"code","source":"lidar_top = lyftdata.get('sample_data', my_sample['data']['LIDAR_TOP']) # selecting LIDAR_TOP out of all LIDARs\nlidar_top","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:20.108685Z","iopub.execute_input":"2023-06-05T07:51:20.109026Z","iopub.status.idle":"2023-06-05T07:51:20.116190Z","shell.execute_reply.started":"2023-06-05T07:51:20.108968Z","shell.execute_reply":"2023-06-05T07:51:20.115216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can use `LidarPointCloud` to read LIDAR data files","metadata":{}},{"cell_type":"code","source":"pc = LidarPointCloud.from_file(Path(lidar_top['filename']))","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:20.118339Z","iopub.execute_input":"2023-06-05T07:51:20.118824Z","iopub.status.idle":"2023-06-05T07:51:20.128847Z","shell.execute_reply.started":"2023-06-05T07:51:20.118747Z","shell.execute_reply":"2023-06-05T07:51:20.127931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pc.points.shape # x, y, z, intensity","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:20.130571Z","iopub.execute_input":"2023-06-05T07:51:20.130963Z","iopub.status.idle":"2023-06-05T07:51:20.140247Z","shell.execute_reply.started":"2023-06-05T07:51:20.130876Z","shell.execute_reply":"2023-06-05T07:51:20.139144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize these lidar points","metadata":{}},{"cell_type":"code","source":"axes_limits = [\n    [-30, 50], # X axis range\n    [-30, 20], # Y axis range\n    [-3, 10]   # Z axis range\n]\naxes_str = ['X', 'Y', 'Z']\n\ndef display_frame_statistics(lidar_points, points=0.2):\n    \"\"\"\n    Displays statistics for a single frame. Draws 3D plot of the lidar point cloud data and point cloud\n    projections to various planes.\n    \n    Parameters\n    ----------\n    lidar_points: lidar data points \n    points          : Fraction of lidar points to use. Defaults to `0.2`, e.g. 20%.\n    \"\"\"\n    \n    points_step = int(1. / points)\n    point_size = 0.01 * (1. / points)\n    pc_range = range(0, lidar_points.shape[1], points_step)\n    pc_frame = lidar_points[:, pc_range]\n    def draw_point_cloud(ax, title, axes=[0, 1, 2]):\n        \"\"\"Convenient method for drawing various point cloud projections as a part of frame statistics\"\"\"\n        ax.set_facecolor('black')\n        ax.grid(False)\n        ax.scatter(*pc_frame[axes, :], s=point_size, c='white', cmap='grey')\n        if len(axes) == 3: # 3D configs\n            text_color = 'white'\n            ax.set_xlim3d([-10, 30])\n            ax.set_ylim3d(*axes_limits[axes[1]])\n            ax.set_zlim3d(*axes_limits[axes[2]])\n            ax.set_zlabel('{} axis'.format(axes_str[axes[2]]), color='white')\n        else: # 2D configs\n            text_color = 'black' # the `figure` is white\n            ax.set_xlim(*axes_limits[axes[0]])\n            ax.set_ylim(*axes_limits[axes[1]])\n        ax.set_title(title, color=text_color)\n        ax.set_xlabel('{} axis'.format(axes_str[axes[0]]), color=text_color)\n        ax.set_ylabel('{} axis'.format(axes_str[axes[1]]), color=text_color)\n            \n    # Draw point cloud data as 3D plot\n    f2 = plt.figure(figsize=(15, 8))\n    ax2 = f2.add_subplot(111, projection='3d')\n    # make the panes transparent\n    ax2.xaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax2.yaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax2.zaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    draw_point_cloud(ax2, '3D plot')\n    \n    plt.show()\n    plt.close(f2)\n    # Draw point cloud data as plane projections\n    f, ax3 = plt.subplots(3, 1, figsize=(15, 25))\n#     f.set_facecolor('black')\n    draw_point_cloud(\n        ax3[0], \n        'XZ projection (Y = 0)', #, the car is moving in direction left to right', ?\n        axes=[0, 2] # X and Z axes\n    )\n    draw_point_cloud(\n        ax3[1], \n        'XY projection (Z = 0)', #, the car is moving in direction left to right',? \n        axes=[0, 1] # X and Y axes\n    )\n    draw_point_cloud(\n        ax3[2], \n        'YZ projection (X = 0)', #, the car is moving towards the graph plane', ?\n        axes=[1, 2] # Y and Z axes\n    )\n    plt.show()\n    plt.close(f)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:20.142129Z","iopub.execute_input":"2023-06-05T07:51:20.142639Z","iopub.status.idle":"2023-06-05T07:51:20.169769Z","shell.execute_reply.started":"2023-06-05T07:51:20.142473Z","shell.execute_reply":"2023-06-05T07:51:20.168799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_frame_statistics(pc.points, points=0.5)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:20.171231Z","iopub.execute_input":"2023-06-05T07:51:20.171537Z","iopub.status.idle":"2023-06-05T07:51:28.971859Z","shell.execute_reply.started":"2023-06-05T07:51:20.171496Z","shell.execute_reply":"2023-06-05T07:51:28.970892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can also use SDK's inbuilt `render_sample_data` to visualize `lidar_top` (remember `lidar_top` is a record in `sample_data` table, just like `cam_front` is )","metadata":{}},{"cell_type":"code","source":"lyftdata.render_sample_data(lidar_top['token'], with_anns=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:28.973429Z","iopub.execute_input":"2023-06-05T07:51:28.973822Z","iopub.status.idle":"2023-06-05T07:51:34.390117Z","shell.execute_reply.started":"2023-06-05T07:51:28.973732Z","shell.execute_reply":"2023-06-05T07:51:34.389178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, above plots show a particular LIDAR sensor (`TOP_LIDAR`), Each sensor is mounted on a ego vehicle aka Autonomous vehicle (AV), here comes `ego_pose`","metadata":{}},{"cell_type":"markdown","source":"## ego_pose","metadata":{}},{"cell_type":"markdown","source":"`ego_pose` contains information about the location (encoded in `translation`) and the orientation (encoded in `rotation`) of the ego vehicle body frame, with respect to the global coordinate system.","metadata":{}},{"cell_type":"code","source":"lyftdata.get('ego_pose', cam_front['ego_pose_token'])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:34.391973Z","iopub.execute_input":"2023-06-05T07:51:34.392288Z","iopub.status.idle":"2023-06-05T07:51:34.399046Z","shell.execute_reply.started":"2023-06-05T07:51:34.392230Z","shell.execute_reply":"2023-06-05T07:51:34.397840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ego_pose parameters are used to bring points from global coordinates system (say bounding boxes' corners) to ego vehicle's frame of reference. As we know that each ego vehicle has a bunch of sensors, each sensor has got some calibration parameters w.r.t ego vehicle","metadata":{}},{"cell_type":"markdown","source":"## caliberated_sensor","metadata":{}},{"cell_type":"markdown","source":"`calibrated_sensor` consists of the definition of a particular sensor (lidar/radar/camera) as calibrated on a particular vehicle. Let us look at an example.","metadata":{}},{"cell_type":"code","source":"lyftdata.get('calibrated_sensor', cam_front['calibrated_sensor_token'])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:34.400504Z","iopub.execute_input":"2023-06-05T07:51:34.400781Z","iopub.status.idle":"2023-06-05T07:51:34.419965Z","shell.execute_reply.started":"2023-06-05T07:51:34.400736Z","shell.execute_reply":"2023-06-05T07:51:34.419093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All extrinsic parameters are given with respect to the ego vehicle body frame. We use caliberate_sensor's parameters to bring points from ego vehicle's frame of reference to sensor's frame of reference. It is used to plot bounding boxes in the images and lidar point cloud.","metadata":{}},{"cell_type":"markdown","source":"We'll need sensor calibration and ego_pose parameters when plotting bounding boxes together with the lidar point cloud. We'll see this later on, before that let's visualize other stuff which `my_sample` has got, starting with bounding box annotations","metadata":{}},{"cell_type":"markdown","source":"# Annotations","metadata":{}},{"cell_type":"markdown","source":"`sample_annotation` refers to a bounding box defining the position of an object seen in a sample.\nAll location data is given with respect to the global coordinate system.\n```\nsample_annotation {\n   \"token\":                   <str> -- Unique record identifier.\n   \"sample_token\":            <str> -- Foreign key. NOTE: this points to a sample NOT a sample_data since annotations are done on the sample level taking all relevant sample_data into account.\n   \"instance_token\":          <str> -- Foreign key. Which object instance is this annotating. An instance can have multiple annotations over time.\n   \"attribute_tokens\":        <str> [n] -- Foreign keys. List of attributes for this annotation. Attributes can change over time, so they belong here, not in the object table.\n   \"visibility_token\":        <str> -- Foreign key. Visibility may also change over time. If no visibility is annotated, the token is an empty string.\n   \"translation\":             <float> [3] -- Bounding box location in meters as center_x, center_y, center_z.\n   \"size\":                    <float> [3] -- Bounding box size in meters as width, length, height.\n   \"rotation\":                <float> [4] -- Bounding box orientation as quaternion: w, x, y, z.\n   \"num_lidar_pts\":           <int> -- Number of lidar points in this box. Points are counted during the lidar sweep identified with this sample.\n   \"num_radar_pts\":           <int> -- Number of radar points in this box. Points are counted during the radar sweep identified with this sample. This number is summed across all radar sensors without any invalid point filtering.\n   \"next\":                    <str> -- Foreign key. Sample annotation from the same object instance that follows this in time. Empty if this is the last annotation for this object.\n   \"prev\":                    <str> -- Foreign key. Sample annotation from the same object instance that precedes this in time. Empty if this is the first annotation for this object.\n}\n```","metadata":{}},{"cell_type":"code","source":"my_annotation = lyftdata.get('sample_annotation', my_sample['anns'][0])\nmy_annotation","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:34.421640Z","iopub.execute_input":"2023-06-05T07:51:34.422191Z","iopub.status.idle":"2023-06-05T07:51:34.432202Z","shell.execute_reply.started":"2023-06-05T07:51:34.422143Z","shell.execute_reply":"2023-06-05T07:51:34.431555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each annotation has all location data with respect to global coordinate system. We will have to take care of rotation, translation and size into account before finalizing the final bounding box coordinates on the sample images luckily we have an inbuilt function for that","metadata":{}},{"cell_type":"code","source":"my_box = lyftdata.get_box(my_annotation['token'])\nmy_box # Box class instance","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:34.433606Z","iopub.execute_input":"2023-06-05T07:51:34.434051Z","iopub.status.idle":"2023-06-05T07:51:34.443310Z","shell.execute_reply.started":"2023-06-05T07:51:34.434007Z","shell.execute_reply":"2023-06-05T07:51:34.442357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_box.center, my_box.wlh # center coordinates + width, length and height","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:34.444717Z","iopub.execute_input":"2023-06-05T07:51:34.444979Z","iopub.status.idle":"2023-06-05T07:51:34.454145Z","shell.execute_reply.started":"2023-06-05T07:51:34.444935Z","shell.execute_reply":"2023-06-05T07:51:34.453495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see our `my_annotation` (remember it is a single object annotation out of the many present in `my_sample`) using the SDK's inbuilt `render_annotation` function","metadata":{}},{"cell_type":"code","source":"lyftdata.render_annotation(my_annotation['token'], margin=10)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:34.459404Z","iopub.execute_input":"2023-06-05T07:51:34.459709Z","iopub.status.idle":"2023-06-05T07:51:39.655156Z","shell.execute_reply.started":"2023-06-05T07:51:34.459660Z","shell.execute_reply":"2023-06-05T07:51:39.654360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each annotation has an `attribute`:","metadata":{}},{"cell_type":"markdown","source":"# Attributes","metadata":{}},{"cell_type":"markdown","source":"An attribute is a property of an instance that can change while the category remains the same.\n Example: a vehicle being parked/stopped/moving, and whether or not a bicycle has a rider.\n```\nattribute {\n   \"token\":                   <str> -- Unique record identifier.\n   \"name\":                    <str> -- Attribute name.\n   \"description\":             <str> -- Attribute description.\n}\n```","metadata":{}},{"cell_type":"code","source":"my_attribute1 = lyftdata.get('attribute', my_annotation['attribute_tokens'][0])\nmy_attribute2 = lyftdata.get('attribute', my_annotation['attribute_tokens'][1])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:39.656728Z","iopub.execute_input":"2023-06-05T07:51:39.657219Z","iopub.status.idle":"2023-06-05T07:51:39.662677Z","shell.execute_reply.started":"2023-06-05T07:51:39.657157Z","shell.execute_reply":"2023-06-05T07:51:39.661532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_attribute1","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:39.664415Z","iopub.execute_input":"2023-06-05T07:51:39.664866Z","iopub.status.idle":"2023-06-05T07:51:39.680881Z","shell.execute_reply.started":"2023-06-05T07:51:39.664801Z","shell.execute_reply":"2023-06-05T07:51:39.679833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_attribute2","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:39.682671Z","iopub.execute_input":"2023-06-05T07:51:39.683369Z","iopub.status.idle":"2023-06-05T07:51:39.692908Z","shell.execute_reply.started":"2023-06-05T07:51:39.683285Z","shell.execute_reply":"2023-06-05T07:51:39.691739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like our subject is standstill","metadata":{}},{"cell_type":"markdown","source":"# Instances","metadata":{}},{"cell_type":"markdown","source":"Object instance are instances that need to be detected or tracked by an autonomous vehicle (e.g a particular vehicle, pedestrian). Let us examine an instance metadata","metadata":{}},{"cell_type":"code","source":"my_instance = lyftdata.get('instance', my_annotation['instance_token'])\nmy_instance","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:39.694643Z","iopub.execute_input":"2023-06-05T07:51:39.695019Z","iopub.status.idle":"2023-06-05T07:51:39.703350Z","shell.execute_reply.started":"2023-06-05T07:51:39.694958Z","shell.execute_reply":"2023-06-05T07:51:39.702452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lyftdata.render_instance(my_instance['token'])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:39.705097Z","iopub.execute_input":"2023-06-05T07:51:39.705665Z","iopub.status.idle":"2023-06-05T07:51:45.258138Z","shell.execute_reply.started":"2023-06-05T07:51:39.705526Z","shell.execute_reply":"2023-06-05T07:51:45.254970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"People generally track an instance across different frames in a particular scene. That's why we have `first_annotation_token` and `last_annotation_token` in `my_instance`","metadata":{}},{"cell_type":"code","source":"print(\"First annotated sample of this instance:\")\nlyftdata.render_annotation(my_instance['first_annotation_token'])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:45.260005Z","iopub.execute_input":"2023-06-05T07:51:45.260681Z","iopub.status.idle":"2023-06-05T07:51:50.604108Z","shell.execute_reply.started":"2023-06-05T07:51:45.260610Z","shell.execute_reply":"2023-06-05T07:51:50.602952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Last annotated sample of this instance\")\nlyftdata.render_annotation(my_instance['last_annotation_token'])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:50.605855Z","iopub.execute_input":"2023-06-05T07:51:50.606208Z","iopub.status.idle":"2023-06-05T07:51:55.717882Z","shell.execute_reply.started":"2023-06-05T07:51:50.606146Z","shell.execute_reply":"2023-06-05T07:51:55.716891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So far we have seen that each sample data point has a bunch of images taken from different cameras, a bunch of lidar data points taken from different lidar sensors, it has annotations for the objects in those images, attributes associated with those annotations. Let's visualize everything which this data point has got in itself using SDK's inbuit `render_sample` function","metadata":{}},{"cell_type":"code","source":"lyftdata.render_sample(token0)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:51:55.719572Z","iopub.execute_input":"2023-06-05T07:51:55.720120Z","iopub.status.idle":"2023-06-05T07:52:18.152342Z","shell.execute_reply.started":"2023-06-05T07:51:55.720061Z","shell.execute_reply":"2023-06-05T07:52:18.151502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3D visualization of a scene","metadata":{}},{"cell_type":"code","source":"my_scene = lyftdata.get('scene',  my_sample['scene_token'])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:18.153728Z","iopub.execute_input":"2023-06-05T07:52:18.154156Z","iopub.status.idle":"2023-06-05T07:52:18.158225Z","shell.execute_reply.started":"2023-06-05T07:52:18.154111Z","shell.execute_reply":"2023-06-05T07:52:18.157508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize `LIDAR_TOP` of all the samples of our `my_scene` in 3D. Here's how we are gonna do this:\n* Each lidar point cloud is in [lidar ranger](https://en.wikipedia.org/wiki/Laser_rangefinder)'s frame of reference, we have to get each point in vehicle's frame of reference. \n* Each bounding box corners are in global coordinate system, we get them to ego vehicles frame of reference using ego_pose parameters. \n* Once we have the lidar point and box corners in vehicle's frame of reference, we plot the points using matplotlib's scatter function and draw the box edges using matplotlib's plot function","metadata":{}},{"cell_type":"code","source":"def get_lidar_points(lidar_token):\n    '''Get lidar point cloud in the frame of the ego vehicle'''\n    sd_record = lyftdata.get(\"sample_data\", lidar_token)\n    sensor_modality = sd_record[\"sensor_modality\"]\n    \n    # Get aggregated point cloud in lidar frame.\n    sample_rec = lyftdata.get(\"sample\", sd_record[\"sample_token\"])\n    chan = sd_record[\"channel\"]\n    ref_chan = \"LIDAR_TOP\"\n    pc, times = LidarPointCloud.from_file_multisweep(\n        lyftdata, sample_rec, chan, ref_chan, num_sweeps=1\n    )\n    # Compute transformation matrices for lidar point cloud\n    cs_record = lyftdata.get(\"calibrated_sensor\", sd_record[\"calibrated_sensor_token\"])\n    pose_record = lyftdata.get(\"ego_pose\", sd_record[\"ego_pose_token\"])\n    vehicle_from_sensor = np.eye(4)\n    vehicle_from_sensor[:3, :3] = Quaternion(cs_record[\"rotation\"]).rotation_matrix\n    vehicle_from_sensor[:3, 3] = cs_record[\"translation\"]\n    \n    ego_yaw = Quaternion(pose_record[\"rotation\"]).yaw_pitch_roll[0]\n    rot_vehicle_flat_from_vehicle = np.dot(\n        Quaternion(scalar=np.cos(ego_yaw / 2), vector=[0, 0, np.sin(ego_yaw / 2)]).rotation_matrix,\n        Quaternion(pose_record[\"rotation\"]).inverse.rotation_matrix,\n    )\n    vehicle_flat_from_vehicle = np.eye(4)\n    vehicle_flat_from_vehicle[:3, :3] = rot_vehicle_flat_from_vehicle\n    points = view_points(\n        pc.points[:3, :], np.dot(vehicle_flat_from_vehicle, vehicle_from_sensor), normalize=False\n    )\n    return points","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:18.159450Z","iopub.execute_input":"2023-06-05T07:52:18.159713Z","iopub.status.idle":"2023-06-05T07:52:18.176185Z","shell.execute_reply.started":"2023-06-05T07:52:18.159668Z","shell.execute_reply":"2023-06-05T07:52:18.175185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_box(box, axis, view, colors, normalize=False, linewidth=1.0):\n    '''Plot boxes in the 3d figure'''\n    corners = view_points(box.corners(), view, normalize=normalize)#\n    def draw_rect(selected_corners, color):\n        prev = selected_corners[-1]\n        for corner in selected_corners:\n            axis.plot([prev[0], corner[0]], [prev[1], corner[1]], [prev[2], corner[2]], color=color, linewidth=linewidth)\n            prev = corner\n\n    # Draw the sides\n    for i in range(4):\n        axis.plot(\n            [corners.T[i][0], corners.T[i + 4][0]],\n            [corners.T[i][1], corners.T[i + 4][1]],\n            [corners.T[i][2], corners.T[i + 4][2]],\n            color=colors[2],\n            linewidth=linewidth,\n        )\n\n    # Draw front (first 4 corners) and rear (last 4 corners) rectangles(3d)/lines(2d)\n    draw_rect(corners.T[:4], colors[0]) #4x3\n    draw_rect(corners.T[4:], colors[1])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:18.178223Z","iopub.execute_input":"2023-06-05T07:52:18.178609Z","iopub.status.idle":"2023-06-05T07:52:18.193201Z","shell.execute_reply.started":"2023-06-05T07:52:18.178526Z","shell.execute_reply":"2023-06-05T07:52:18.192245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def draw_3d_plot(idx, lidar_token):\n    '''Plot the lidar + annotations on a 3D figure'''\n    # sample lidar point cloud\n    lidar_points = get_lidar_points(lidar_token)\n    points = 0.5 # fraction of lidar_points to plot, to reduce the clutter\n    points_step = int(1. / points)\n    pc_range = range(0, lidar_points.shape[1], points_step)\n    lidar_points = lidar_points[:, pc_range]\n    \n    # Get boxes, instead of current sensor's coordinate frame, use vehicle frame which is aligned to z-plane in world\n    _, boxes, _ = lyftdata.get_sample_data(\n        lidar_token, flat_vehicle_coordinates=True\n    )\n    fig = plt.figure(figsize=(15, 8))\n    ax = fig.add_subplot(111, projection='3d')                    \n    point_size = 0.01 * (1. / points) # size of the dots on plot\n    ax.set_facecolor('black')\n    ax.grid(False)\n    ax.scatter(*lidar_points, s=point_size, c='white', cmap='gray')\n    for box in boxes:\n        c = np.array(lyftdata.explorer.get_color(box.name)) / 255.0\n        plot_box(box, ax, view=np.eye(3), colors=(c, c, c), linewidth=0.5)\n    ax.set_xlim3d(-40, 40)\n    ax.set_ylim3d(-40, 40)\n    ax.set_zlim3d(-4, 40)\n    \n    # make the panes transparent\n    ax.xaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax.yaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax.zaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))\n    ax.set_title(lidar_token, color='white')\n    filename = 'tmp/frame_{0:0>4}.png'.format(idx)\n    plt.savefig(filename)\n    plt.close(fig)\n    return filename","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:18.194982Z","iopub.execute_input":"2023-06-05T07:52:18.195288Z","iopub.status.idle":"2023-06-05T07:52:18.213817Z","shell.execute_reply.started":"2023-06-05T07:52:18.195239Z","shell.execute_reply":"2023-06-05T07:52:18.212689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir tmp # a temporary folder to contain plot jpegs","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:18.215474Z","iopub.execute_input":"2023-06-05T07:52:18.215787Z","iopub.status.idle":"2023-06-05T07:52:19.372516Z","shell.execute_reply.started":"2023-06-05T07:52:18.215738Z","shell.execute_reply":"2023-06-05T07:52:19.371187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's take a quick look at the 3d Plot\nfirst_sample_token = my_scene['first_sample_token']\nsample = lyftdata.get('sample', first_sample_token)\nlidar_token = sample['data']['LIDAR_TOP']\nfilename = draw_3d_plot(0, lidar_token)\nImage.open(filename)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:19.374529Z","iopub.execute_input":"2023-06-05T07:52:19.374909Z","iopub.status.idle":"2023-06-05T07:52:26.773404Z","shell.execute_reply.started":"2023-06-05T07:52:19.374843Z","shell.execute_reply":"2023-06-05T07:52:26.772296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize `my_scene` in the form of a video","metadata":{}},{"cell_type":"code","source":"frames = []\nfirst_sample_token = my_scene['first_sample_token']\ntoken = first_sample_token\nfor i in tqdm(range(my_scene['nbr_samples'])):\n    sample = lyftdata.get('sample', token)\n    lidar_token = sample['data']['LIDAR_TOP']\n    filename = draw_3d_plot(i, lidar_token)\n    frames += [filename]\n    token = sample['next']\n#     break","metadata":{"execution":{"iopub.status.busy":"2023-06-05T07:52:26.775140Z","iopub.execute_input":"2023-06-05T07:52:26.775710Z","iopub.status.idle":"2023-06-05T08:07:46.394845Z","shell.execute_reply.started":"2023-06-05T07:52:26.775655Z","shell.execute_reply":"2023-06-05T08:07:46.394065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clip = ImageSequenceClip(frames, fps=5)\nclip.write_gif('pcl_data.gif', fps=5);","metadata":{"execution":{"iopub.status.busy":"2023-06-05T08:07:46.396117Z","iopub.execute_input":"2023-06-05T08:07:46.396549Z","iopub.status.idle":"2023-06-05T08:08:01.808407Z","shell.execute_reply.started":"2023-06-05T08:07:46.396503Z","shell.execute_reply":"2023-06-05T08:08:01.807213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"\nfrom IPython import display","metadata":{"execution":{"iopub.status.busy":"2023-06-05T08:08:01.810248Z","iopub.execute_input":"2023-06-05T08:08:01.810680Z","iopub.status.idle":"2023-06-05T08:08:01.816862Z","shell.execute_reply.started":"2023-06-05T08:08:01.810607Z","shell.execute_reply":"2023-06-05T08:08:01.815753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open('pcl_data.gif','rb') as f:\n    display.Image(data=f.read(), format='png')","metadata":{"execution":{"iopub.status.busy":"2023-06-05T08:08:01.818604Z","iopub.execute_input":"2023-06-05T08:08:01.819012Z","iopub.status.idle":"2023-06-05T08:08:02.130123Z","shell.execute_reply.started":"2023-06-05T08:08:01.818935Z","shell.execute_reply":"2023-06-05T08:08:02.125816Z"},"trusted":true},"execution_count":null,"outputs":[]}]}