Building autonomous vehicles with (almost) no labels: a deep dive with Dr. Holger Caesar

The development of autonomous vehicles is a complex process, requiring a deep understanding of robotics, computer vision, and machine learning. One of the most challenging aspects of creating these high-tech vehicles is the development of perception systems that allow the vehicle to 'see' and understand its surroundings. Holger, an Assistant Professor in the Intelligent Vehicles group of TU Delft in the Netherlands, recently shared his insights on this topic during a fascinating presentation at Kognic.

Holger: a pioneer in autonomous vehicle perception

Holger's research interests are in the area of Autonomous Vehicle perception and prediction, with a particular focus on scalability of learning and annotation approaches. He is best known for developing the influential autonomous driving datasets nuScenes and nuPlan, as well as his contributions to the real-time 3D object detection method PointPillars.

Before joining TU Delft, he worked as a Principal Research Scientist at Motional, an autonomous vehicle company. There, he started teams that focused on Data Annotation, Autolabeling, and Data Mining, gaining valuable experience in the practical aspects of building autonomous vehicles.

TU Delft’s self driving car prototype

The evolution of perception systems

Holger's presentation centered on the evolution and future direction of perception systems for autonomous vehicles. He divided the development of these systems into three generations:

1. The first generation relied heavily on manually labeled datasets and object detectors trained with full supervision. However, this approach was costly and inflexible, necessitating a more efficient approach.
2. The second generation sought to automatically label vast amounts of driving data. Although this allowed robotaxis to expand to new cities quickly, the underlying offline perception system still required significant human effort for training.
3. Holger envisions a third generation that would require little to no human labeling. Techniques like active learning, object discovery, self-supervised learning, and foundation models could reduce human annotation efforts significantly. These methods would also make it easier for existing vehicles to adapt to new cities and new robotics applications, such as bikes, trains, and boats.
   
   

The Intelligent Vehicles Group at TU Delft

Holger is part of the Intelligent Vehicles Group at TU Delft. This team of six professors is dedicated to exploring various aspects of autonomous driving, including perception, localization, mapping, and planning. They aim to advance perception technology, implement cutting-edge deep learning AI methods for prediction and planning, and investigate the challenges posed by alternative transportation methods.

One of their notable initiatives is the nuScenes dataset, a pivotal resource for autonomous vehicle perception research. By providing annotated data from real-world environments, the nuScenes dataset has catalyzed innovation and collaboration within the autonomous driving community.

The road ahead

Looking forward, Holger and his team acknowledge the evolving landscape of autonomous vehicles and the pressing challenges that lie ahead. Issues such as scalability, cost-effectiveness, and weather adaptability pose significant hurdles that demand innovative solutions and multidisciplinary collaborations.

In their ongoing work, Holger's team is also focusing on learning how to build safer bicycles and safer cities for cycling. They are developing a sensor-equipped bicycle to gather a unique dataset of bicycle behavior, collected on the crowded bicycle paths of Delft in the Netherlands.

In conclusion, the team's commitment to autonomous research underscores a dedication to pushing the boundaries of technology and shaping the future of transportation. Their collaborative ethos and forward-thinking approach position them at the forefront of autonomous driving innovation, driving towards a safer, more sustainable future on the roads.

Special thanks to Holger and his team for the fantastic research and presentation.