505 Jobs

The Big-Data All-Rounder

IAV measurement data platform can process vast data volumes – if need be, in real time and spread across the globe

The volumes of data involved in automotive engineering are multiplying. Added to this are new services which, during the vehicle’s lifetime, are also reliant on up-to-date information – such as from the Internet of Things. With its tried and proven measurement data platform, IAV offers a solution that can manage virtually unlimited data volumes and, in conjunction with an IAV analysis tool, analyze and visualize them. Customers can use it as a software-as-a-service and, if necessary, draw on their own solutions for analysis.

Particularly large data volumes are generated in fleet testing where the OEMs can either use SSD memories and flash cards in the vehicles or wireless links (e.g. LTE, UMTS or WiFi) for communicating them to the computing center. In addition, there is information from test benches and, to an increasing extent, also data from the world of the Internet of Things, such as weather information. The vast quantities of bytes are securely transferred, managed and analyzed as part of automotive development or used as the basis for providing new services.

IAV has over 15 years of experience in this field, and all of this has been channeled into the measurement data platform. The largest entity at the moment acts as a private cloud which IAV operates at three different locations in Gifhorn to ensure absolute resilience to failure. Large data volumes are part of everyday routine: “In some fleet tests, we clock up many terabytes a month“, reports Björn Steffen, Senior Vice President for Mobility at IAV. “But with our current memory capacity of two petabytes, that isn’t a problem – particularly as we can scale up into the exabyte range.”

Over 20 data logger formats

The data are fed into the system via storage medium or the Internet, with secure SSL
links being used for long-distance transmission. “Besides standard data formats, over 20 data logger raw formats can be analyzed with our system”, says Jens Schulze, head of the Fleet Support and Data Analytics department at IAV. “This means we can connect everything that needs to speak to the car. If there is anything missing, we can easily integrate it.” The raw data are initially fed into the object memory and can be processed in computations (for example, in databases or as JSON or XML documents) and also filed in a big data cluster. The raw data, however, remain completely intact in order to address new matters arising in the future, for example.

The analysis tools run on the application server which spreads the computing load over several hundred nodes, making it possible to process data volumes running to terabyte level every hour. As such, over 500 CPUs and more than 4.5 TB of RAM are currently used for analyses. This volume can come from fleet tests or ongoing customer operations – depending on a given meteorological situation, for instance, this would make it possible to send out weather warnings to drivers.

Together with hardware, also developed at IAV, the overall system is used as well in customer vehicles not only for analyzing telemetry data but also up-to-date fault memory entries in the control units as a way of providing customers with direct assistance at short notice. The system also provides a live view into the vehicle. Depending on the application case, data can be selected and displayed on the platform directly from the vehicle.

Real-time and application-oriented analyses possible

Basically speaking, the system is capable of carrying out real-time and application-oriented analyses. Real-time analyses demand a slight latency between question and answer, and they use a relatively high amount of memory and computing capacity. With application-focused analyses, on the other hand, the question is known in advance, containing the demand on resources.

A clearly structured presentation of results is provided by a web front end or automatically generated report documents capable of handling various formats, such as point clouds, histograms or heat maps and which every customer can match to its needs and corporate identity.

Decentralized computing clusters at local level

“Our measurement data platform is an open system that all customers can use to suit their own needs”, Steffen says, summarizing. “It is scalable and provides tremendous capacity by parallelizing computations.” Even if data are generated abroad, there is no need for customers to compromise on performance or security because IAV operates its own “remote zones” for this purpose. “These decentralized computing clusters are responsible for storing and analyzing data”, Schulze says. “But system control remains in Germany.” This means it is only necessary to transfer analysis results and not the far greater quantity of measurement data. The entire system can even be operated in a private customer cloud or public cloud.