Automakers Eye Up Control of the OS

Software now controls almost everything on a modern vehicle.

From obvious functions like the infotainment system, to less obvious things such as brakes and then to the mundane, such as electric windows, software is the lifeblood of the car, the thing that makes it tick.

The majority of the code in vehicles today is going towards powering the autonomous functions and interacting with the on-board sensor hardware, enabling cars to change lanes or drive in traffic with little or no human input required. A report from McKinsey last year claimed cars had up to 150M lines of code, an increase of 15 times over the claimed 10M from 2010.

However, contrary to the claims of huge increases in lines of code, Tim Wong from Nvidia told TU-Automotive that it’s actually neural networks doing a lot of the heavy lifting in today’s autonomous vehicles. “So believe it or not, it’s not increasing lines of code, because nowadays, a lot of the software stack is based on deep neural networks. So, deep neural networks use training data to come up with the network that you then run for influence inside the car, thus it’s not so much about sequential code anymore, although it is still there. Now it’s a lot about using parallel networks, one for lane detection, one for object detection, sign detection, sign understanding and so on.”

Since the car’s software stack now touches almost everything. The stack, incidentally, comprises the entire platform car runs on, as Dan Cauchy, executive director of Automotive Grade Linux, explains. “So it starts with the low-level software, which interacts and supports the on-board hardware. Then there’s the middleware, which applications use for common functions. The next layer is the application framework; controlling the whole life-cycle of an application, and the security framework, to stop a rogue application accessing something it shouldn’t. Finally, the API framework, for speech recognition, navigation, telephony, radio, which is the most important thing, as apps need them to function.”

Running the incredibly powerful, complex software means performance becomes a big focus. “The challenge of running multiple neural networks on a computer platform is the performance of that computer platform,” Wong said. “Running any slower than 30 frames per second, you come into safety issues, because the car isn’t computing where it needs to be quick enough. Performance is always a constraint, so the algorithms need to be optimized. We can accelerate the performance of our neural networks which has become very important lately but it is and always will be a challenge.”

With the key role being played by software in today’s vehicles, it’s easy to forget about the computing hardware inside the car, which has to keep up with the advances made in machine learning and neural networks. “We’ve definitely seen companies hit the upper limits of hardware,” Wong said. “Then what they do is they add a second hardware platform. So if you have a platform which can do 320 trillion operations per second, we have customers hitting that limit and needing a second, so they can do 640 trillion operations per second. This means the two are basically load sharing, moving the algorithms between the two. However, power is a concern, especially in electric vehicles, since more hardware requires more power which reduces range.”

Wong also told us that despite that incredibly powerful hardware, software is the key differentiator. “What we’re seeing from our customers is that they want to be able to use a variety of hardware but the software needs to be different. So, some companies are using radar and cameras, some are using LiDAR, some ultrasonic. The hardware that’s chosen in each specific case makes the software solution different. We provide our software platform as full source, so customers can modify it if they want. Some, however, want to write their own custom code, so we take out our standard module and let them install their own. They do this because, for example, a sports car company is going to want its car to drive differently when compared to a vehicle from a luxury car company. So, while the hardware all achieves the same thing, allowing the car to be controlled by the computer, the software changes what that control looks like.”

What, then, are the benefits of automakers developing their own software solutions? Why not just outsource it to an experienced software developer? “It gives the automaker control over its own destiny and allows them to invest in their own unique platform for their long term future,” Cauchy said. “They can decide which feature they want to enable or disable, with no back and forth between a software supplier, which can be very inefficient and take time to find a resolution. Instead, it can be done all in house.”

Wong, meanwhile, saw it a little differently. “The automakers are concerned about liability,” he said. “By bringing it in house, they can fully understand the algorithms and can test it in their own way. This helps them feel better about liability. If I’m going to do something right, I want to control it, and the way to do that is to bring it in house. That’s the way they see it.”

With the technical advances in hardware and software, plus the strategic moves of automakers to bring software development in-house, it’s easy to forget that we’re only just scratching the surface of autonomous vehicles, with the next step having a vehicle understand what another vehicle could do, according to Wong. “So object perception is fairly well understood today,” he said. “Cars can perceive objects and obstructions in front of them, whether that be a child, animal, tree branch, another vehicle. It’ll be fine tuned but there probably won’t be massive changes.

“What will improve is the object understanding and potential for bad behavior. So if a car is about to turn in front of you, or you are passing a line of parked cars, we as humans know it’s possible the car in front might cut you off, or a car door will open. An autonomous vehicle doesn’t understand this though. So trying to get that understanding of what an object might potentially do, before it happens, is where the work is currently going.”

Finally, security and testing is a big concern for any software developer. Since the auto software industry is relatively new, this can cause issues, as Thaddeus Bender from HackerOne told us: “From a development criteria, vehicles don’t align with common criteria that we’re used to. There’s no common security standards in automotive. Each automaker has their own standards that they test to, but there’s no agreed upon approach to achieve a desired level of security.

“However, the bigger problem is that, unlike computers or phones, these vehicles will be on the road for decades. Currently, there isn’t a good, cost efficient, effective way to maintain them. Automakers have previously had the mindset of ‘we build something, release it, sell it, then forget about it’. With complex on-board software and therefore constant security concerns, this mindset needs to change. I do see some understanding that there needs to be constant maintenance with over the air updates but I don’t think that we’re at a level where they understand that they must actually monitor and maintain privacy concerns.”

Leave a comment

Your email address will not be published. Required fields are marked *