Taking Driver Monitoring Systems to a New, Feature-Rich Level

Nov 06, 2024

Imagine if your car could recognize your face directly and allow only the owner to drive it. Or maybe its safety systems could go beyond merely detecting driver inattention and dangerous in-car phone usage to noticing medical conditions and even emotions. These are some of the emerging features provided by advanced Driver Monitoring Systems (DMSs), with much more to come as software-defined vehicles (SDVs) develop.

Welcome to Season 3, Episode 4 of “Get In: The Software-Defined Vehicle Podcast from BlackBerry.” This series explores the possibilities created by — and technologies behind — the revolution in global transportation we are witnessing today.

In this fourth episode of Season 3, we visit the Consumer Electronic Show® (CES) 2024 to talk to Michael Horani, Director, Advanced Engineering at Mitsubishi Electric, about how his company is at the forefront of DMS, with the help of BlackBerry® QNX® and BlackBerry IVY®.

Click to watch the full podcast episode or keep reading for excerpts.

“Mitsubishi Electric is a huge conglomerate,” says Horani. “The company is split into different business units. Mitsubishi Electric Automotive, America, falls under the automotive division. My team specializes in innovation.” The Mitsubishi booth at CES® 2024 had some very exciting technology on show. “This is the fruit of the last couple of years of collaboration between Mitsubishi Electric and BlackBerry, both with the QNX and IVY products.” Mitsubishi’s CES demonstration focused on safety. “It highlights the possibilities in the cabin to make the ride safer and more entertaining while reducing the stress of the driver.

“Most of the features focus on driver monitoring of health, biometrics, tracking heart and breathing rates, and where you're looking,” explains Horani. Astronauts and elite athletes normally get this type of monitoring, but drivers don’t. Now they do. It's almost like giving them superpowers. “We also look at the occupants of the vehicle to see if seatbelts are buckled. For that we use various algorithms and sensors. We also detect objects on the road so we can determine if the surface is slippery, if it's safe to drive, and whether that will impact your braking distance.”

BlackBerry IVY collects and normalizes data from sensors throughout the car to make features like this possible. “A perfect example is how it enables an algorithm to generate insights about what's happening in the cockpit, such as gaze detection,” continues Horani. “This tracks where you look. That normally would have taken us between 9 to 12 months to develop. With the great team that BlackBerry has collaborating with us, we were able to complete this in less than two months. The promise of IVY has delivered well.”

Mitsubishi also had new contactless technology to share. This is heavily focused on reducing the distraction and stress on the user when they're trying to access the vehicle.

Artificial intelligence (AI) has been a significant enabler of these functions, and Horani sees even more mature platforms arriving next year for CES 2025. “AI has been a huge theme in the automotive industry this year,” he says. “It's going to get even wider deployment and adoption in the future. We have several examples of generative AI coming to the vehicle, and not necessarily just in infotainment. At the core of it is safety, but it will also enable us to move from Level 2 to Level 3 to Level 4 autonomous vehicles. The motor vehicle will understand what the occupants are doing and turn on an autonomous mode to take over control from the driver.”

Mitsubishi showcased its sophisticated DMS features at CES 2024. On the company’s stand, Kyle Troutt, Senior Hardware Engineer at Mitsubishi Electric, demonstrated sensing technology aimed at occupant monitoring created through its innovation studio Filament Labs. “The technology on show is a combination of radar and overhead camera information, pulling in data to assess where occupants are seated in the vehicle and give an overview of their body size,” says Troutt. The system uses radar, millimeter wave data, and thermal information to sense biometric information such as breathing rate.

Occupant sensing for both rows of seats provides features like seatbelt usage detection, but the DMS delivers more driver-focused information. This starts with recognizing the identity of who is driving. “We can use that recognition information to do all sorts of things,” Troutt explains. “This ranges from loading regular personal information and setting up your Human-Machine Interface to your preferences to giving control for vehicle owners over who else might be driving their car and what they are allowed to do.”

But the DMS can go well beyond this and even detect driver emotions. “We have the capability to determine a neutral or a more negative face, to pull that information in to determine levels of distraction,” says Troutt. “An angry driver is not a safe driver. We can detect smoking based on hands near the mouth, phone usage, and yawning because a tired driver is also an unsafe driver.

“One of our most important highlights this year is biometric information,” continues Troutt. “We have heart rate detection, which is completely contactless. It's based on data from the vehicle occupant’s face indicating blood flow. We also have breathing rate information from the radar, and then temperature information from our thermal sensors. Overall, this assesses the medical state of the driver to determine if there's an emergency that might not be detected from posture collapse. The driver’s overall thermal profile could also provide data used for optimizing AC control.

“We also detect passenger interaction, using data from microphones,” says Troutt. “If you're having a normal conversation with somebody else in the car, you're probably driving safely. But if you have a car full of kids being loud, it's probably very stressful. We can also detect distraction based on interaction. If you're looking at your screen and touching it, you're probably not paying very much attention to your driving.”

All of these features are built on top of BlackBerry IVY. “These algorithms — whether driver and occupant monitoring focused, or generally biometric sensing — are pooled into one mega application,” Horani explains. “We have nine different synthetic sensors combined to decide if the driver is distracted. It's very complicated to determine this using just one or two factors like whether your eyes are dilated or your heartbeat. The more data we have, the better it is to make that determination. BlackBerry has been instrumental in making this feature work.”

“There are several things we can extract from this,” continues Horani. “If a distraction has been detected, we have a timeline of an event. It will take a photo or create a log.” Mitsubishi’s technology can also decide what to do based on the type of distraction. “We can suggest different ways to handle this, whether it's just a coffee stop, or listening to a specific music and changing lighting to calm the driver down. If the distraction is too high and comes from noise, we can address it by turning on focus mode.

“In focus mode, the screen becomes more limited,” says Horani. “Touch input is restricted. You can open apps, but it helps the driver to reduce distraction, such as incoming text messages. Once the distraction and the stress level are calmed down, we can exit focus mode and go back to normal operation.”

Mitsubishi Electric’s system can also access external sensors to provide additional functionality, which was one of the most intriguing features showcased at CES. “We have the capability to determine the face ID of the driver sitting in the cockpit, but we're also able to determine the direction of the head and the eyes,” confirms Horani. “We've taken that outside of the vehicle and created completely contactless entry, both establishing that you are allowed to have access to the car and giving you a very specific gesture that allows you to open the trunk automatically without accidentally triggering it.” Features like this are going to change the face of how we interact with our vehicles — quite literally — by making interaction facial, all enabled by sophisticated DMSs such as Mitsubishi’s powered by BlackBerry IVY and QNX.