How to Know an Autonomous Driver Is Safe and Reliable?

The race to deploy fully autonomous vehicles (AVs) is accelerating. Waymo has already reached over 250k trips per week while Tesla and Zoox are ramping up. The key question for scaling is not “Can AVs drive?” but “How to know AVs are safe and reliable at scale?” 

As developers, we live by a simple creed: if it’s not tested, it’s broken. We write unit tests, integration tests, and end-to-end tests to gain confidence. But what happens when the 'test environment' is the unpredictable chaos of a public road, where an edge case can have severe repercussions?

This is the monumental challenge facing autonomous vehicle (AV) engineers. Traditional testing methods struggle to verify complex AV behavior that depends on AI making tons of real-time decisions. This problem requires moving beyond familiar testing paradigms into a new frontier of validation. In this article, I explore the multi-layered evaluation strategy to answer the critical question about the readiness of the AV driver.

The Crucial Role of Evaluation: A Challenge on Par with Development

In my experience, evaluation of AV driving is a challenge of similar complexity to the development of the driver. A lot of rigor, innovation, and investment are needed for confident validation given the challenges below.

• Really high bar for evaluation quality: Thorough evaluation is imperative. The lives of road users depend on safe and reliable operation. This means the bar for evaluation quality is really high.
• Complexity of real-world scenarios: AVs navigate the infinite variability and unpredictability of the real world. This includes complex traffic patterns, adverse weather, and unpredictable behavior of road users.
• The "long tail" of edge cases: AVs may perform well in common driving scenarios; however, ensuring good behavior in rare "edge cases" requires extensive testing.
• Lots of data and compute: Evaluating AVs requires vast amounts of real driving data and significant resources for simulation and analysis.
• Need for statistical significance: Demonstrating safety requires not just anecdotal evidence but statistically significant data across a vast range of conditions. Only a measurable outcome can help drive software development in the right direction. 
• Need for closed-loop evaluation: AV system’s real test is in closed-loop environments where the AV's actions (e.g., braking) influence other actors (e.g., the car behind slows down for a braking AV). Building such a dynamic simulation is very challenging, as it needs realistic reactions. Open-loop tests (where agents don’t react to AV) are useful but not sufficient.

Hence, we need a comprehensive approach to evaluation. Let’s talk about various techniques that help build such a powerful evaluation system. 

Key Evaluation Methods 

Figure: Multi-stage AV evaluation framework from simulation to real-world deployment.

1. Targeted Scenarios for Simple Tests: Building Blocks of Confidence

Targeted scenarios are focused on assessing specific functionalities. They act as the building blocks verifying the AV's ability to handle fundamental driving tasks. These could be made of synthetic scenarios or segments from on-road driving. Some examples include:

• Lane keeping: Evaluating the ability to accurately stay within lane markings on straight and curved roads.
• Adaptive cruise control: Testing the AV's response to a lead vehicle, ensuring smooth deceleration and a safe following distance.
• Traffic light recognition and response: Assessing the ability to identify traffic light states and execute appropriate actions.
• Basic obstacle avoidance: Evaluating the capability to detect and safely maneuver around static obstacles.

2. Targeted Scenarios for Edge Cases: Stressing the System's Limits

Evaluating the AV's resilience to rare and challenging "edge cases" is crucial for ensuring safety in unpredictable real-world environments at scale. The scenarios push the boundaries to help prevent edge case failures. These are generated synthetically or from real-world disengagements (with additional challenges injected). Examples include:

• Unexpected pedestrian behavior: Testing the response to jaywalking, pedestrians coming out of occlusion, or behaving erratically.
• Adverse weather conditions: Evaluating performance in heavy rain, snow, fog, and strong winds that could significantly impact perception.
• Complex incident scene: Assessing the vehicle's ability to navigate a complex scene with construction and emergency vehicles.

3. Unbiased Large-Scale Resimulation for Validation: The Power of Data Replay

Unbiased large-scale resimulation plays a critical role in gaining a statistically significant understanding of the AV's overall performance. This involves replaying vast amounts of real-world driving data through a high-fidelity simulation environment. The key aspect here is the "unbiased" nature of the data. The data is from the real-world driving logs capturing the inherent complexities enhanced by fuzzing in various parameters to utilize the expensive on-ground data better. This allows for:

• Identifying systemic weaknesses: Uncovering potential issues due to the sheer volume and diversity of the simulated scenarios, including potential regressions.
• Quantifying performance metrics: Accurately measuring key performance indicators (KPIs) such as improper stops, safety-critical events, and road rules across a wide range of situations.
• Validating software updates: Rigorously testing new software versions against a consistent and representative dataset.

4. Biased Large-Scale Simulation: Exploring the Long Tail of Risk

Biased large-scale simulation focuses on testing scenarios that are known to be challenging. This approach allows developers to proactively explore the "long tail" of critical events. Since they require less compute, I have seen individual teams benefit from targeted large-scale simulations. Such a system allows for:

• Targeted situations at scale: Based on an automated way to identify challenging situations (e.g., presence of fire trucks) for testing specific ability at scale using on-ground data.
• Inject sensor or computer failure: Large-scale simulation with sensor or compute failures to test the fallback mechanisms.
• Parameter sweeping: Systematically varying simulation start times, road user reactions, and other relevant factors to create a lot more situations from the existing data.

5. On-Road Validation With a Human Driver: The Importance of Real-World Interaction

Real-world testing remains an indispensable part of the evaluation process. Conducting tests with a human safety driver allows the AV to test the end-to-end performance. Key objectives of on-road testing with a safety driver include:

• Gathering real-world data: Collecting valuable data on the AV's performance in diverse and unscripted situations, and testing the whole system, including hardware and off-board issues.
• Identifying unexpected challenges: Uncovering cases not anticipated during simulation.
• Building public trust: Demonstrating the AV's capabilities in the real world under controlled supervision.
• Validating simulation results: Comparing the AV's on-road behavior with the predictions from the simulation helps improve simulation techniques.

6. On-Road Validation Without a Human Driver: The Final Frontier

The ultimate stage involves on-road validation without a safety driver. This happens after gaining very high confidence in AV’s safety and reliability. There are aspects that can only be tested in a fully autonomous setting.

Beyond simply demonstrating capability, driverless on-road validation is crucial for uncovering and addressing issues that might be masked by the presence of a human safety driver. A human might instinctively take over, preventing the system from truly exercising autonomy and revealing limitations. Examples of aspects that can only be truly tested in fully autonomous mode include:

• Efficacy of remote assistance: When the AV encounters a situation it cannot handle, it may request remote human assistance. Only in a driverless mode can the seamless hand-off, the clarity of information provided to the remote operator, and the effectiveness of the remote intervention be truly validated.
• Drivers might be taking over earlier: Specific scenarios requiring immediate action might prompt a human safety driver to intervene prematurely. This masks the AV's true capability in handling niche but critical situations.
• Issues with real users: The nuances of pulling over to pick up or drop off passengers in urban environments often present unexpected challenges that only manifest in autonomous mode.
• Operational aspects: The entire operational chain that activates in the rare event of an AV stranding to safely retrieve the vehicle and manage the situation can only be validated in fully autonomous testing.

Key objectives of driverless on-road validation include:

• Demonstrating system maturity: Providing evidence that the autonomous system can safely navigate real-world environments without human intervention.
• Gathering data on unsupervised operation: Collecting valuable data on the AV's performance and interactions in a fully autonomous mode, revealing true system limitations and capabilities.
• Building public acceptance: Demonstrating the potential of fully autonomous vehicles to operate safely and efficiently in everyday traffic, fostering trust, and enabling wider adoption.

Challenges with Simulation-Based Techniques

As one would imagine, on-road testing is expensive and may still not catch all “edge cases.” Hence, there is a lot of focus on making simulation-based techniques more powerful. Below are some challenges with the current techniques and how AI is accelerating the progress. 

• Simulation realism: Creating environments to accurately reflect the complexities of the real world is a challenge. Moreover, subtle differences between simulation and reality, such as pose divergence, can lead to inaccurate evaluation.
• Creating quality metrics: Defining meaningful and reliable metrics that can be automated is complex. 
• Butterfly effect: Seemingly insignificant changes in initial conditions can lead to drastically different outcomes in AV behavior, making it difficult to pinpoint the root cause.
• Human triage: A significant amount of human review and analysis is needed to identify potential issues.
• Statistical significance for tail events: Gathering enough data to achieve statistical significance for rare "edge cases" is extremely challenging.
• Simulating full system: Accurately simulating all aspects of the system, including user inputs (e.g., opening the door), remote assistance interventions, and off-board calls, is difficult.

The Role of AI in Improving Evaluation

I am quite bullish on AI playing an increasingly vital role in overcoming the evaluation challenges. We are already seeing promise in some interesting and high-priority areas:

• Realistic synthetic scenarios: GenAI can be used to generate novel and challenging synthetic scenarios. These scenarios can specifically target known weaknesses or explore the "long tail" more efficiently.
• Better sensor simulation: AI, trained on vast datasets of real sensor data (camera, lidar, radar), can generate more realistic and accurate sensor outputs. This includes handling pose divergence and modeling noise for credible virtual testing. 
• Powerful metrics: AI can be trained to identify subtle patterns and anomalies in AV behavior that might be missed by traditional metrics. This can lead to the development of more nuanced and predictive safety metrics offering a better understanding of the AV's capabilities and potential failure points.

Conclusion: Multi-Layered Evaluation Is Necessary for AVs

AVs represent one of the most complex engineering challenges of our generation. While the public sees the vision of autonomous cars, it's the meticulous and unseen work of evaluation that makes it a reality. This journey is not a single sprint but a rigorous marathon. Neither simulation nor road testing alone is sufficient. 

True confidence is born from a comprehensive framework that systematically builds proof from edge cases in targeted simulations to validate holistic performance on public roads. As systems evolve with increasingly powerful AI, this disciplined approach will be critical in not just building an autonomous driver but in building a verifiable system of trust.