No Amount of AV Test Miles Predicts Every Scenario

While autonomous driving is expected to save lives being better than human drivers, AVs will probably be held to a higher standard than human-piloted cars.

It’s not a technical problem so much as a social one and it comes down to lawyers, says Sandy Munro, CEO of Munro & Associates, an engineering and manufacturing consulting firm. “That’s where the real problem lies. It’s not so much in the technology being good for the general population but in how many lawyers will get a chance to make a ton of money based on car accidents,” he says.

Ideally, regulators, and lawyers, would be satisfied if there was a positive risk balance compared to the performance of a human driver, that is, if it was safer to get into an AV than into a conventional car. “The hardest sell about AVs is sometimes the expectation that a driverless vehicle will be perfect, never get into an accident in any scenario and any time,” says Jack Weast, a senior principal engineer at Intel and the vice-president of automated vehicle standards at Mobileye. “If we start at 40,000 deaths and get down to 40, that’s incredible improvement. Of course, one death feels like too many but if the goal is zero, and you have to prove it’s zero, we probably won’t see this technology in our lifetime, even if it would save millions of lives.”

Weast is co-author of Safety First for Autonomous Driving, a 2019 report that provides an overview of the steps for developing and validating a safe automated driving system, based on the input of carmakers, tiered suppliers and key technology vendors.

Toward a single AV certification

With the process of homologation, there’s a range of standards and regulations for conventional vehicles and a robust market of services that can guide automakers through the process for wherever they want to sell their cars. However, while carmakers have come to terms with these for conventional vehicles, the complexities of an autonomous vehicle would make the process much more onerous.

In Europe, automakers must obtain type approval for a new model, that is, have national authorities certify that a model of a vehicle meets all EU safety, environmental and production requirements before it can be sold in the EU market. Yet, there is no type approval in place today for autonomous vehicles.

In the US, automakers self-certify vehicle safety, although they can apply for a rating from National Highway and Transportation Safety Administration’s (NHTSA), New Car Assessment Program (NCAP). In October 2019, NHTSA said it would propose updates to the NCAP program, some of which relate to new, although unspecified, technologies.

The United Nations Economic Commission for Europe (UNECE) administers road safety conventions, including ADAS systems, for Europe and its World Forum for Harmonization of Vehicle Regulations (WP 29) aims for globally set regulations on vehicles. Again, work is underway but nothing has been released.

This is where industry has an obligation to help, Weast says. After all, governmental bodies seldom have engineers or scientists on their teams. “We believed the fastest way is to roll up our sleeves and help create and lead new standards to benefit everybody, rather than complaining that governments aren’t moving fast enough.”

Is simulation testing the key to AV homologation? 

Any certification process, whether global or local, will demand more comprehensive testing than anything we’ve seen so far. TÜV SÜD, an independent third-party testing organization, estimates there could be as many as 100 million situations for each automated driving function that would need to be tested.

Road testing of AVs has limitations, says Matt Cragun, Nvidia’s senior manager for automation and training. Individual components, like automatic emergency braking, can be tested on a track or public road but with autonomous driving, the capabilities are much more complex. “The only way you’ll be able to test and have confidence an autonomous vehicle is safe is to test against many types of situations in a controlled way. You can’t expect a physical road test to do all that,” Gragun says.

In addition, if a situation is dangerous, the safety driver must take over to prevent a crash. So, there’s no way of knowing whether an autonomous system would actually identify and stop for, say, a bicyclist darting across its path in the rain. “You can’t test situations that are too risky or uncontrolled,” Cragun says.

Simulation has long played a role in training self-driving systems and it can play a role in safety testing as well, according to Cragun. Nvidia is collaborating with TÜV SÜD and AVL, a specialist in developing, simulating and testing powertrain systems, to develop virtual certification methods and solutions for autonomous vehicles.

While the total number of miles driven is one of the basic metrics for AVs, Danny Shapiro, senior director of automotive for Nvidia, says: “Miles aren’t what’s important. It’s the scenarios. Simulation gives us the ability to do that. More important is being able to standardize the simulation so that everyone is testing against the same database.”

The Nvidia/AVL/ TÜV SÜD collaboration began in 2018 and includes the development of requirements for the homologation of simulations themselves. That’s done, according to Cragun, by matching what the simulated physics of a vehicle to its physical counterpart. Does a simulated vehicle respond exactly like the physical one when the brakes are applied, for example? If so, engineers can be confident that the simulation is adequate for testing.

MIT, in collaboration with the Toyota Research Institute, recently announced its own simulation system to train driverless cars. MIT says its Virtual Image Synthesis and Transformation for Autonomy (VISTA) uses a small dataset captured by humans driving on a road to synthesize a photo-realistic world with an almost infinite number of new viewpoints from trajectories that the vehicle could take in the real world.

A fine balance

While the pace of government is slow, neither Weast nor Cragun thinks a lack of homologation standards for self-driving cars is slowing down development of the technology. “In general, the industry would like to see a regulatory framework much sooner rather than later. They don’t want to be blocked if the technology is ready but the regulatory framework is not,” Weast says.

Cragun adds: “Certainly, there is some lack of clarity in what future will hold in terms of regulation. At same time, industry is very cognizant that poor regulation could stifle the industry. … In some ways, I’m happy with the pace.”

Leave a comment

Your email address will not be published. Required fields are marked *