Focus on health

Cars will observe their occupants in the future

02.06.2023  — 

IAV has developed an AI that uses a camera to estimate the vital signs of vehicle occupants and can thus check the driving ability of all persons inside the vehicle. The technology will increase road safety and prevent accidents in the future.

The Car that Cares; 210309 Bild2 TCTC V03 jrr sRGB; IAV E CARES 01 OHNE DISPLAY Hires JPG.jpg
As soon as the camera and AI detect critical heart and respiration rates, the car could be brought to a controlled stop.

Modern vehicles are standardly equipped with interior cameras that can capture both visual and (near infrared) NIR modalities. Currently, they are installed to monitor the driver and thereby prevent accidents. In addition, there is also the possibility of using them to estimate the vital functions of the occupants.

Vital signs estimation avoids accidents

In the future, the vital sign estimation could detect critical events such as sudden heart attacks, strokes or fatigue at an early stage. It can then also detect whether drivers or passengers are impaired, e.g. under the influence of drugs or alcohol.

This is particularly useful if the intelligent vehicle prevents an impaired driver from driving the vehicle. This significantly reduces the risk of a fatal accident.

Estimation of vital parameters detects micro sleep in time

A modern car analyses the vital signs and can assess whether the driver is in a phase of drowsiness where micro sleep is very likely. The camera can use the algorithm to detect changes in heart rate and breathing that typically occur during this phase. The system concludes how likely it is that the phenomenon of micro sleep may occur in the near future.

It is precisely in this phase that intelligent assistance systems such as lane departure warning systems can be useful in reducing the risk of accidents caused by micro sleep.

The algorithm estimates the vital signs: IAV derives the respiration from the chest area and the PPG signal from the face. From this, heart rate and respiration rate can be determined. Note: The video shows a prototype and is for visualisation purposes. The signals in the video may differ from those of the occupants.

Health care with data from the vehicle

Many people stay in their cars often and for long periods of time without moving much. This allows smart vehicles to record and monitor their vital parameters, or changes in heart rate and breathing, over the long term and with greater accuracy than when they are moving. In contrast to a smartwatch, which you move a lot in everyday life, the prediction is much more accurate when you are at rest.

This can be used in preventive medicine. Changes in breathing as well as heart rate allow conclusions to be drawn about the current state of fitness and can predict the risk of developing cardiovascular diseases such as cardiac arrhythmias, heart attacks and asthma.

Vehicle interior as examination room

The vehicle has the potential to act as a substitute for a doctor’s traditional waiting room, for example, when the patient waits in their car while their current and long-term vital signs are transmitted to the practice via camera. The doctor can then decide whether an in-person appointment is necessary, prescribe medication or adjust medication.

While it is unlikely that smart vehicles will completely replace doctor’s visits in the future, it is quite possible that long-term monitoring of vital signs could minimise many in-person visits.

Safety is a must, well-being a need

Vital parameter estimation can also help to increase the well-being of the occupants by inferring emotions and activating comfort functions in the vehicle depending on the situation.

This has many advantages over the algorithms used so far, which focus exclusively on facial expressions. This is because they cannot reliably estimate emotions if people wear a beard or are skilled at hiding their emotions.

Vital signs are different. They distinguish the emotional state more reliably from the behavior of heart rate and breathingand estimate fairly accurately whether people seem focused or are angry.

Heart rate estimation with RGB NIR camera and Deep Learning

To estimate heart rate, IAV uses specific regions of the face that are usually uncovered and criss-crossed with numerous arteries and veins. This allows Deep Learning to use a camera pointed at the face to capture subtle colour and brightness variations that occur when the arteries pulsate, causing the blood volume to change.

More specifically, it works via the photoplethysmographic signal (PPG). Arterial blood absorbs light more efficiently than surrounding tissue, and when the arteries expand due to the pulse, the absorption rate of the light changes. The camera detects this change and maps it to a PPG signal.

Measurement of respiratory rate with optical flow method

In order to estimate the respiratory rate, IAV used the optical flow method. Here, an optical flow algorithm detects chest movements in the video images and assigns them to a specific respiratory rate. The respiration rate is derived from the movement of the thorax and upper abdomen during the inhalation and exhalation phases. In this way, this movement can be assigned to a respiratory signal, which is recognised as a person’s respiratory rate.

IAV presents topic at InCabin in Brussels

Patrick Laufer will again take part as a speaker at the InCabin exhibition on 21.06.2023. In his presentation, he will discuss whether a single RGB NIR camera is sufficient to determine the heart and breathing rates of all occupants in a vehicle. He will also highlight possible problem cases.

Are you interested in this topic? Then also read the article on camera-based estimation of height, weight and gender of vehicle occupants.

For more information, please contact Patrick Laufer on LinkedIn or at

patrick Laufer Heart Rate
Patrick Laufer is a development engineer and doctoral student in the Vehicle Safety Engineering department at IAV.