Environment perception and understanding represents one of the most challenging problems in autonomous driving. Many actions taken by the autnomous vehicle rely on the results of the perception layer which might be considered as one of the most critical subsystems in an autonomous vehicle. In order to improve the reliability of percpetion, multiple sensors are used in the vehicles as well as in the infrastructure. The reliability of perception might further be increased by fusing all perception related data coming from either vehicles or infrastructure which stands for the main aim of the performed experiment described in this report.

We have performed environment perception related tests under real circumstances by deploying infrastructural sensors (cameras, LiDARs) together with vehicle sensors (cameras, LiDARs, dGPS) in a parking lot of the university suitable for testing. Let us give a brief overview of the sensors used during this experiment:

In the infrastructure two separate nodes have been deployed, both equipped with cameras and a 16 channel LiDAR of type Velodyne VLP-16. On top of that we have included also a vehicle into the system which was equipped with a 64 channel LiDAR of type OUSTER OS1-64 and 7 cameras (3x front, 2x side, 2x rear). For data processing the NVidia DRIVE PX2 platform has been used (see Figs. 2 and 3).

The goal of the experiment was to test 2D and 3D synchronized detections in such a distributed sensory environment as well as their tracking and their fusion. In camera images the detections appear in form of 2D bounding rectangles while in the LiDAR pointcloud in form of 3D bounding boxes (see Fig 1b). All these detections have been transformed into a common reference coordinate system. In order to obtain the necessary transformation matrices the whole distributed sensory system had to be calibrated first. Besides calibration, another important aspect was the time synchronization between the sensors. During the experiment we relayed on GNSS as the time source for synchronization. To test the time synchronization in different sensory setups a test system was developed shown by Fig. 3.

Both the reliability and detectability play crucial role in safety critical perception systems. Objects in such a distributed environment might be detected by multiple sensors (attached to the infrastructure or vehicles), thus higher reliability and coverage might be achieved. By exchanging detection related information between vehicles, the sensed area might significantly be extended. During the experiment the objects of interest were pedestrians and vehicles, however other type of objects can be included into the system, as well. 

As it was already mentioned, the system was calibrated, thus all the detections could be transformed into a global coordinate system, which was also necessary to be able to inject all the detections into the virtual model of the test area. All the detections automatically appeared also in the virtual environment together with the included measurement vehicle (see Fig. 1a). The final result of the experiment might be considered as a digital twin of the test area containing all static and dynamic objects present in the real environment. If an object changes its state in the real environment, this change is automatically reflected in the virtual model, as well.

The building blocks of the realized system stand for the basis of a so-called central perception system being under development in our Lab, where the vehicles as well as the infrastructure exchange perception related data through a central system. Since the system must be capable of real-time performance, the throughput, the computing performance as well as the complexity of algorithms, the time synchronization are key factors to be considered. Such a central perception system may take us closer towards the realization of the digital twin of road networks, including all static and dynamic traffic participants.

‍

Video available at:

‍

‍

‍

‍