DEVELOPER BLOG

Synthetic Data Cloud for real-time Driver Monitoring System (DMS) development and validation of Vision AI models

By SKY ENGINE AI   01 September 2022

Driver Monitoring

|

Occupant Monitoring

|

Automotive

|

Synthetic Data

|

Deep Learning


SUBSCRIBE
       

SKY ENGINE AI driver monitoring with synthetic data and computer vision for car, truck, lorry, train, airplane interior

 

Driver monitoring systems (DMS) based on the AI and Computer Vision (CV) are quickly becoming standard on board of vehicles of any type and model. The safety standards push the sensitivity and alerting adequacy to play critical role when assessing the DMS's applicability to broad adoption. The drivers do not need to face additional distraction coming from misleading alerts as their attention should be laser focused on the driving task. Although there are several apprentice DMS systems on the market yet, the gap still exists but the good news is that it can be filled by the deep learning in virtual reality with Synthetic Data where the AI models are tested and trained prior to the real-world deployment leading to improved adherence to the regulations, better interoperability with ADAS systems, and most importantly thousands of fatalities can be prevented.

 

Time has come to invest in safety improvement and the AI-based Driver Monitoring System

The Driver Monitoring System (DMS) will become a standard safety component throughout the world as early as 2023. In the United States the Department of Transportation will compel to begin developing guidelines and rules to combat distracted and intoxicated driving, as well as to modernize the New Car Assessment Program (NCAP). Initially, European regulations will apply to distracted and sleepy driving. Eventually, Europe will need impairment detection systems to cover alcohol and drugs.

There is always time to consider improvements in safety through this technology. According to the National Highway Traffic Safety Administration, distracted driving was responsible for over 3,000 deaths in 2019, a 10% increase over 2018. Drunk driving has also grown. In 2019, 10,142 people were killed in drunk driving accidents. Early projections for 2020 show a 9% rise in DUI fatalities. Many people anticipated road deaths to decrease during the epidemic, but the reverse happened.

To further boost the reliability up to higher Safety Levels it is critical to develop scalable, complementary and interconnected AI models with high adequacy. This can be mostly ensured by training sensors in a virtual environment prior to the real-world deployment. The SKY ENGINE AI platform enables Data Scientist friendly development of such AI models using platform's deep learning in virtual reality technology for computer vision applications. Driver monitoring systems usually employ RGB or infrared (IR) cameras and computer vision software to guarantee that the driver is awake and aware, and is paying attention to the road. The system may be configured to do a sequence of consecutive actions, beginning with a mild alert or warning and advancing to slowing or halting the vehicle if the driver is no longer able to control it.

SKY ENGINE AI driver monitoring with synthetic data and computer vision for car, truck, lorry, train, airplane interior

Key benefits of the DMS/OMS for manufacturers, fleet owners, insurers, drivers and occupants 

The key benefits of using the driver and occupants monitoring system include:

  • Increased safety of the driver, passengers and others on the road by activities, emotions, attentivness and drowsiness detection
  • Alerting driver before collisions happen – avoid collisions than report on after the fact
  • Compliance to General Safety Regulations (GSR)
  • Improved comfort and experience by driver and occupants identification for settings personalization with i.e. facial recognition
  • Supports autonomous driving L3+
  • EuroNCAP 4-5 star rating relevance
  • Helps drivers improve their skills – especially important for newer drivers
  • Track and confirm drivers following the regulations and procedures in fleet safety policy
  • Highly decrease operational costs, reduce insurance claims and expenditure, reduce collisions
  • Cost-efficient system based on scalable, modular approach and seamless integration with exisiting systems (ADAS)
  • High flexibility of adaptation to the universe of models, markets, and vehicle interiors
  • Driver's position, weight and pose monitoring (3D skeleton) to facilitate optimization of the safety systems

Such DMS can be developed at a fraction of cost of real data acquisition and labelling providing complete solution for driver monitoring and in-cabin sensing. In addition, it is viable to employ low-cost cameras in the car as synthetic data and complex vision AI algorithms can be created in the SKY ENGINE AI platform to allow reliable driver monitoring.

For the human driver, the scene is too complex, reactions too slow and timeframe too short to provide quick and adequate reactions in any conditions. The combination of the DMS with Advanced Driver-Assistance Systems (ADAS) can use predictive analytics to further boost crash-prevention capabilities. ADAS incorporates technology like as automated emergency braking, blind-spot recognition, and lane departure alerts and its capabilities may be further enhanced when integrated with the DMS. For instance, after detecting drowsiness or high distraction level the DMS can send signal to the ADAS to increase the distance to preceding vehicle first before stopping the car completely.

The computer vision AI models for the Driver Monitoring System (DMS) that can be developed and effortlessly retrained (when necessary) in the SKY ENGINE AI platform:

  • Identification of the driver in order to allow the vehicle to automatically restore its preferences and settings
  • Monitor driver fatigue and alert him when potential drowsiness situation is detected
  • Monitor driver attentiveness by ensuring he’s keeping his eyes on the road, hand on the wheel and that he/she is aware of any dangerous situation
  • Pilot an user interface using the eyes by automatically selecting HMI areas, highlight important features on the road, etc.

In addition, training the AI models for the DMS in the SKY ENGINE AI platform preserves the privacy as these models are trained in virtual reality with synthetic data: humans, cars, objects, etc. Virtual vehicle interiors serve the purpose of training, testing and validation of the AI models before deployment in a real car saving hundreds of hours and highly reducing the overall cost of such DMS.

Meet our Synthetic Drivers and Occupants in Context

The SKY ENGINE AI Synthetic Data Cloud enables creation of very realistic humans and objects to propel development of next generation in-cabin monitoring systems. Meet Tina and Mark, who have been created, along with other hundred drivers, for the job of simulating the driver and the occupant behaviour in the car, performing several activities that may also be prohibited by law (depending on the country and regulatory), but should be accurately detected by any modern monitoring system like the DMS. These synthetic humans were created along with the entire car interior and context under changing outside environment conditions to preserve the impact they have on the trained vision AI models.

 

SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS – Complex ground truth system with semantic masks and 3D key points
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS with 2D/3D bounding boxes
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS – Complex ground truth system with depth maps
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS with instance masks

Figure 1 – Simulated RGB synthetic data for in-cabin monitoring with highly complex pixel-perfect ground truths generated in the SKY ENGINE AI cloud, 8 images: (Left, top-bottom) Meet Tina – Driver and Occupant; Driver and car interior – camera in the center stack rear view mirror; SKY ENGINE AI complex ground truth system – semantic segmentation masks for separated body parts and 3D key points including invisible face landmarks; Ground truth – depth map; (Right, top-bottom) Meet Mark – Driver; Driver and car interior – camera in the console; Complex ground truth system – 2D bounding boxes; Semantic segmentation masks.

SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS
SKY ENGINE AI synthetic data for AI models training for driver monitoring system - DMS ADAS

Figure 2 – Synthetic data of car interior, driver and objects (i.e. bag, keys, mobile phone, etc.) and ground truths simulated and generated in the SKY ENGINE AI platform, 6 images: (Left) Drowsy male driver (RGB), Drowsy male driver (IR), Semantic segmentation, (Right) Keypoints and gaze estimation, Distracted female driver, Depth map.

Driver-monitoring systems capable of detecting drowsiness or distraction are only the beginning. As these technologies develop, they will become a part of a larger interior sensing platform that offers personalization, improved safety, infotainment, and even communication with smart home systems. The DMS can recognize the driver and enable personalisation to change the seat, temperature, side mirrors, and so on to the driver's preferences. The technologies will be able to determine if the driver is intoxicated or experiencing a medical emergency. Drivers will be able to operate some functions in the car by using their eyes or gestures and  the camera can watch both the driver and the cabin, allowing it to recognize whether a kid has been left in a car seat, assess whether an essential object has been forgotten, or assist in personalizing infotainment, HVAC, or other in-cabin capabilities. But to enable all these functionalities with unprecedented accuracy it is required that the vision AI models tasked to perform detection and recognition of multiple actions taken by the driver and occupants including capturing changing situations in the interior of the car have to be rigorously trained with myriads of samples of well-balanced and diverse data. Such datasets can be simulated and generated in different modalities in the SKY ENGINE AI cloud. 

Figure 3 – Driver gesture recognition AI model training with synthetic data (video). (Left) Infrared synthetic data, (Right) 3D pose estimation (keypoints and skeleton).

 

Universe of applications: insurance, cab, lorry, train, airplane fleet

The SKY ENGINE AI platform can effectively be used to simulate Physically-accurate, photo realistic synthetic data in the RGB and IR modalities enabling accelerated training of the accurate AI models in the driver monitoring applications. These computer vision models, when incorporated in the DMS/OMS system, enable real-time assistance in identifying in-car activities and help recognizing event risks, such as their probability, direct cause, likelihood of potential injuries and severity accumulated over pre-defined period of time. Accidents can be predicted, managed, and, at best, largely prevented. Insurers may be advised of the most likely events, given on-the-road and in-cabin circumstances, with a high level of accuracy. This can assist in discussing policies and applying discounts to the drivers or fleet owners reinforcing strict safety measures, noticing less issues, and recommending additional training assistance. Such system can apply driver scoring for post-accident analysis of the behaviour to further optimize insurance policy and its cost, which can be especially interesting for fleet owners. In the premium offering, drivers can get a monetary value in the form of monthly cash-backs, based on their road safety performance measured by accurate evidence-based risk indicators for insurance companies.

The DMS is expected to keep all the data aboard the vehicle to avoid triggering privacy concerns. However, to dispute any insurance claims it could be employed to work with recording mode on. Fleet owners or insurance companies can get insights on the detailed status of the driver and can alert a dispatch center or the driver to deliver real-time life-saving alerts. Insurers can offer Pay-How-You-Drive (PHYD) policy models to enable accurate evidence-based risk indicators including drivers scoring. Next, the drivers can get a monetary value in the form of monthly cash-backs or policy discounts, based on their road safety performance.

Let us know about your cases and get access to the SKY ENGINE AI platform or get tailored AI models or synthetic datasets for your driver and occupant monitoring applications. As we support much more industries than just an automotive a broad range of data customization is available even for specific sensors and environments.