In order for autonomous vehicles to operate safely in the real world, they must be able to adapt to a multitude of changing conditions, navigating curves, crossing bridges and climbing hills.Before advanced driver assistance systems (ADAS) or fully self-driving vehicles hit the road, they undergo a lengthy period of testing where the vehicle’s sensors and artificial intelligence are tested in a variety of simulated real-world environments — exactly what the Microsoft team behind Project Road Runner does.

Using photo-realistic simulation and deep learning to train autonomous driving algorithms, the team gathers data and trains AI platforms through real-world simulation.

The core team, comprising lead program manager Aditya Sharma and lead engineer Mitchell Spryn, deploy skills ranging from robotics, deep learning and hardware engineering to cloud computing and development operations to create photo-realistic driving environments.

“Cars take up a lot of space in our lives and in our cities. Most of a car’s lifetime is spent sitting around unused, wasting space,” Aditya Sharma, lead program manager of Project Road Runner, wrote in a company blog post. “Making cars autonomous means more space in our cities, less accidents on the roads and a cleaner and greener environment.”

Microsoft’s AirSim, an open-source simulation platform for drone research, proved particularly useful to the Road Runner team — the photo-realistic environments, built using the Unreal Engine, are ideal for training autonomous driving algorithms.

A recently unveiled simulation environment inspired by the popular Hana Highway in Maui winds through rainforests, ocean front and grassy mountains with tunnels and bridges along the way — the team’s upcoming lane detection tutorial also uses the Hana Highway environment to train deep learning models.

“Simulation has become the backbone of the autonomous driving industry, providing a means to collect extensive amounts of data for model training as well as providing a safe testbed to crash-test these models,” Sharma wrote. “The closer our simulated scenarios are to reality, the closer we get to making autonomous cars a part of our lives.”

The Road Runner team has also been involved in Microsoft’s efforts in the AV space with external partners, including auto supplier Bosch, as part of the OpenADx initiative, which also involves several other players like Mathworks, Red Hat and the Eclipse Foundation.

The team recently released the Autonomous Driving Cookbook, an open source collection of scenarios, tutorials and demos to help developers learn about various aspects of the autonomous driving pipeline.

Tutorials in the cookbook are presented as Jupyter notebooks and come with their own datasets, helper scripts and binaries.

While the tutorials leverage popular open source tools — including Keras or TensorFlow — as well as Microsoft open source and commercial technology, such as AirSim and Azure virtual machines, the primary focus is on the content and learning.

“A big motivation behind the Cookbook for us is to promote this spirit of collaboration and so, we welcome other experts in the area to contribute tutorials to the cookbook to make it a go-to resource for people who wish to enter and grow in this exciting field,” Sharma wrote in a February blog post.

— Nathan Eddy is a filmmaker and freelance journalist based in Berlin. Follow him on Twitter @dropdeaded209_LR.