Virtual testing provides repeatable measures of performance, speeds development, and lowers the risk inherent to real-world driving activities. Simulation is one example of our virtual testing.
For example, if we’re working on the way our software handles pedestrian crosswalks, we can pull from our database of interactions for each occasion where our Driver encountered a pedestrian at a crosswalk. Then, in simulation, we can replay those interactions and evaluate how the new code would handle not only this situation, but myriad permutations of it. We can change the parameters of the encounter: Are there two adult pedestrians? An adult and a child? Is it a group of pedestrians? This allows us to test the Driver against a diverse set of cases without needing to drive this scenario repeatedly in the real world, hoping we encounter all of the interesting variations we care about.
The vast majority of our simulation experiments are short, and focus on specific interactions, allowing us to efficiently cover a huge number of effective testing miles.
The Concept of Determinism
Adapting game engines or other applications to create simulated environments for self-driving vehicle software to navigate is effective in the short term — it gets testing operations up quickly. But this approach has an important drawback, which involves the concept of determinism.
In simulation, a test that’s deterministic is one that, given the same environmental inputs, provides the same result. No randomness is involved. Game engines can be remarkable pieces of software and we’ve been able to make use of their technology in the design and preparation of individual elements for our simulations. But they’re not built specifically for self-driving car testing. They often don’t run in lock-step with the autonomy software leading to results that are not deterministic. Given the same sensor inputs, simulations based on game engines will not necessarily provide the same test result.
That’s tricky for simulation testing because it introduces a level of uncertainty. Simulation testing aims to verify the robustness of new software code. Did the test fail because of the new code? Or did it have something to do with the non-determinism of the simulation environment? When it’s based on game engines, it’s difficult to be certain. With our systems, the simulation and autonomy software move in lock step — removing the uncertainty.
Aurora’s Offline Executor
We’ve engineered a tool built specifically for the validation of self-driving car software — the offline executor — and it’s crucial to our simulation efforts. It uses the same programming framework as the Aurora Driver software stack, and the same libraries. In fact, because it is purpose-built to work with our system, the offline executor allows us to have deterministic offline testing for any module, for any team, across the entire organization. That’s a huge benefit. Otherwise, integrating game engines to work with self-driving software can sometimes feel like putting a square peg into a round hole.
With our offline executor, we don’t have to spend a huge amount of effort integrating the simulation environment with the rest of our system. Instead, we have a system designed from the ground up that enables lock-step execution of self-driving software and the simulation it interacts with. We can then test the same software that runs on our vehicles at massive, cloud-scale including detailed simulation of latencies and compute delays that happen on real-world hardware.
When we build a new capability, the first step is to build the simulations which feature the interaction we’re developing. Next, we write the code, and then we run the code through the simulations using the offline executor.
Using the offline executor streamlines our development process by employing software modules in a manner similar to the overall self-driving stack. For example, when testing an element of the motion-planning module, the offline executor’s simulation module feeds the planner the set of inputs it otherwise would get from the perception module. These are synthesized inputs, but the motion-planning module can’t tell the difference. That means the system reacts in the same way to a simulated environment as it would in the real world — which is crucial to making reliable autonomy software.
Here’s an example: The simulation module feeds the motion-planning module the knowledge that a car is passing in the right lane, and a pedestrian is traversing a crosswalk up ahead.
Based on these inputs, the motion-planning module will calculate a trajectory — the vehicle’s intended course over the next few seconds. That output goes to the validation module. The validator’s job is to decide whether the trajectory is a good one. Did the motion-planning module do the right thing? The validation module evaluates the trajectory by asking a series of questions. Did the trajectory obey the law? Did the motion planning module meet its objective? Is this trajectory comfortable for vehicle occupants? If the motion-planning module’s trajectory passes all such tests, it passes the interaction.