Our diverse Virtual Testing Suite is a major accelerator that enables us to make rapid development progress.
Virtual testing refers to any time that our software is running offline in response to synthetic or real historical data as opposed to operating in real-time on the road. As we’ve said since our early days, a strong virtual testing program is a marker of a mature self-driving effort, and we take a multifaceted approach to thoroughly assess the performance of each element of our system before it reaches our vehicles. This combined with thoughtful on-road testing will allow us to deliver the Aurora Driver safely and quickly at scale.
Why we test off-road
Decades of collective experience writing production-ready software has taught us that mature organizations test early and often. Virtual testing is the pillar of our development process because it allows us to:
- Test more thoroughly. It’s practically impossible to experience every possible scenario in real life. Virtual tests allow us to test different permutations of the same scene in a controlled way. It’s also not feasible to run thousands of tests on the road to evaluate a single change to the codebase, but we can do that off-road.
- Find and address issues early. On-road tests examine how all parts of our system work together, but virtual tests allow us to determine how well tiny parts of the system are working individually. This makes it much easier to identify the root cause(s) of problems early in development.
- Fine-tune new capabilities quickly. Fast, reliable feedback from virtual tests enables rapid iteration and improvement.
- Develop objective measures of progress. We run the same tests with the same parameters each time we make a change to the codebase. When a new version passes more tests than its predecessors, we can say it’s performing better.
- Make on-road testing safer and more efficient. When our software passes thousands of off-road tests, we can be confident that our Driver will perform well on-road. Virtual testing also allows us to streamline our real-world tests to focus on the areas where we need additional feedback.
Put simply, virtual testing has been and will continue to be critical to successfully developing and deploying the Aurora Driver.
Aurora’s Virtual Testing Suite
We’ve invested — and continue to invest — in the infrastructure that enables our multifaceted Virtual Testing Suite to run millions of off-road tests per day.
A robust virtual testing program requires this complementary suite of tests, beyond just simulation, that assess how software works at every level. Thus, our Virtual Testing Suite includes a mix of codebase tests, perception tests, manual driving evaluations, and simulations.
Codebase unit and regression tests
We don’t leave testing to the end of the development process. As code is written, we write both unit tests (e.g., seeing if a method to calculate velocity gives the right answer) and integration tests (e.g., seeing whether that same method works well with other parts of the system). New work must pass all relevant tests before we can merge it with the larger codebase, allowing us to identify and fix issues early.