Modernizing SCADA Systems and OT/IT Integration With Apache Kafka

SCADA control systems are a vital component of IT/OT modernization. The old IT/OT infrastructure and SCADA system are monolithic, proprietary, not scalable, and miss open APIs based on standard interfaces. This post explains the modernization of such a system based on the real-life use case of 50Hertz, a transmission system operator for electricity in Germany. Two common business goals drove them: Improve the Overall Equipment Effectiveness (OEE) and stay innovative. A lightboard video about the related data streaming enterprise architecture is included.

The State of Data Streaming for Manufacturing in 2023

The evolution of industrial IoT, manufacturing 4.0, and digitalized B2B and customer relations require modern, open, and scalable information sharing. Data streaming allows integrating and correlating data in real-time at any scale. Trends like software-defined manufacturing and data streaming help modernize and innovate the entire engineering and sales lifecycle.

I have recently presented an overview of trending enterprise architectures in the manufacturing industry and data streaming customer stories from BMW, Mercedes, Michelin, and Siemens. A complete slide deck and on-demand video recording are included.

This blog post explores one of the enterprise architectures and case studies in more detail: Modernization of legacy and proprietary monoliths and SCADA systems to a scalable, open platform with real-time data integration capabilities.

What Is a SCADA System? And How Does Data Streaming Help?

Supervisory control and data acquisition (SCADA) is a control system architecture comprising computers, networked data communications, and graphical user interfaces for high-level supervision of machines and processes. It also covers sensors and other devices, such as programmable logic controllers, which interface with process plants or machinery.

Data streaming helps connect high-volume sensor data from machines, PLCs, robots, and other IoT devices. This is possible in real-time at scale with stream processing. The de facto standard for data streaming is Apache Kafka and its ecosystems, like Kafka Stream and Kafka Connect.

Enterprises leverage Apache Kafka as the next generation of Data Historians. Integrating and pre-processing the events with data streaming is a prerequisite for data correlation with information systems like the MES or ERP (that might run at the edge or more often in the cloud).

50Hertz: A Cloud-Native Scada System Built With Apache Kafka

50hertz is a transmission system operator for electricity in Germany. The company secures electricity supply to 18 million people in northern and eastern Germany.

The infrastructure must operate 24 hours, seven days a week. Various shift teams and a mission-critical SCADA infrastructure supervise and control the OT systems.

50hertz next-generation Modular Control Center System (MCCS) leverages a central, scalable, event-based integration platform based on Confluent.

The first four containers include the Supervisory and Control (SCADA), Load Frequency Control (LFC), and Time Series Management and Forecasting applications. Each container can have multiple services/functions that follow the event-based microservices pattern.

50hertz provides central governance for security, protocols, and data schemas (CIM compliant) between platform containers/ modules. The cloud-native 24/7 SCADA system is developed in the cloud and deployed in safety-critical edge environments.

50hertz presented their OT/IT and SCADA modernization leveraging data streaming with Apache Kafka at the Confluent Data in Motion tour 2021. Unfortunately, the on-demand video recording is available only in German. Therefore, in another blog post, I wrote more about the case study: "A cloud-native SCADA System for Industrial IoT built with Apache Kafka."

Lightboard Video: How Data Streaming Modernizes SCADA and OT/IT

Here is a five-minute lightboard video that describes how data streaming helps with modernizing monolith and proprietary SCADA infrastructure and OT/IT environments:

If you liked this video, make sure to follow the YouTube channel for many more lightboard videos across all industries.

Apache Kafka Glues Together the Old and New OT/It World

The 50Hertz case study showed how to modernize an existing legacy infrastructure with cloud-native technologies, whether you deploy at the edge or in the public cloud. 

A common question in these scenarios is the proper communication and integration protocol when you move away from proprietary legacy PLCs and OT interfaces. MQTT and OPC-UA established themselves as excellent standards with different sweet spots. Data Streaming with Apache Kafka is complementary, not competitive. 

How do you leverage data streaming in your manufacturing use cases? Do you deploy at the edge, in the cloud, or both? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.