O-Net LiDAR Viewer

The goal of the project is to not only develop an application to gather and display point cloud data from LiDAR sensor, but also demonstrate the Object Detection feature implemented using PointPillars Pytorch model. We created our PointPillars model based on Open3D-ML's implementation and serialized it into TorchScript format and then loaded into our C++ object detection module for x86_64 platform. While for Xilinx UltraScale+ MPSoC platform, we took advantage of Xilinx Vitis AI Model Zoo. The LiDAR Viewer project largely depends on Open3D, which is a very cool open-source library that provides a full stack of features to create an application quickly. Open3D uses Dear ImGUI for the UI widgets and Google's Filament for the 3D rendering.

LiDAR Viewer depends on other three projects that are developed by O-Net, cppbase, LidarDevice and PCDetector. LidarDevice is responsible for communications with LiDAR sensor, saving and loading gathered point cloud data. PCDetector on the other hand is responsible for object detections using pre-trained ML models. For x86_64 platform, PCDetector is directly linked into LiDAR Viewer and run model locally. While for Xilinx ZCU104 platform, PCDetector runs as a RPC receiver, and LiDAR Viewer runs on PC sending point cloud and getting inference results through RPC channels. (Figure 1. Module Diagram)

The following demo shows LiDAR Viewer running through a webrtc server, which can also run as a standalone desktop application. It loads KITTI dataset point cloud and sends it to Xilinx ZCU104 dev board, which runs PCDetector behind a RPC receiver and sends inference results back to LiDAR Viewer, which then updates the visualizer to show the point cloud and object detection results.