How Industry 4.0 is driving HMI technology

in English

In an industrial context, a human machine interface (HMI) allows an operator to interact with their machine or process. It shows the machine’s status, and allows the operator to input commands.

But an HMI can be implemented in many ways, using hugely varying levels of technology.

The second industrial revolution – which lasted from 1870 until about 1970 – saw the arrival of mass production using electrical power. In this environment, HMIs comprised display lamps, analogue panel meters, switches, and pushbuttons. However, the advent of the third industrial revolution, from about 1970 onwards, meant that production became automated. Operators had to understand and react to the status of large numbers of plant items, and how they were interacting with one another in real time.

This meant that representing each plant variable with discrete devices like switches and lamps was no longer practical, as the numbers involved would quickly exceed both the control panel’s and the operator’s capacity. In any case, the amount of information provided was severely limited, and, above all, the control panel design was totally inflexible.

The response was to marshal all the plant sensor outputs onto a CRT or, as they became available, a flat panel display. Operators could respond to displayed plant status using a keyboard and mouse, or, more recently, a touch screen. Note that this did not entirely eliminate discrete input and output (I/O) components, which in fact are still have a role today alongside the more advanced HMI solutions.

Yet digitalisation, and the growth of Industry 4.0 since around 2014, meant that simple flat panel solutions in turn became inadequate. For various reasons which we will explore, Industry 4.0 environments generate vast amounts of data, which have to be presented to operators in readily-assimilated formats. They have to be given information rather than data. Operators also have to respond in more nuanced and sophisticated ways.

This all means that although most HMIs are still based on flat panel displays, their hardware, software and processing power are vastly different to those of, say, 20 years ago. And flat panel systems are no longer the whole story; some applications are now benefitting from Augmented Reality (AR) and Virtual Reality (VR) technologies.

Accordingly, we will now look at the nature and volume of data generated by Industry 4.0 installations, why this presents a challenge to HMI design, and the software and hardware solutions now available to address this challenge. We also review the more recent trends related to AR and VR, and their impact on HMI design.

Industry 4.0 and its high data volumes

The term Industry 4.0 was first introduced in 2011 by German scientists to promote the advances in AI and manufacturing automation over earlier technologies . Also referred to as the fourth Industrial Revolution, 4IR, or smart manufacturing, it builds on the inventions of the Third Industrial Revolution—or digital revolution—which brought us computers, more functional electronics, the Internet, and much more. Industry 4.0 brings these developments beyond the previous realm of possibility with several foundational types of disruptive technologies that can be applied all along the value chain :

Connectivity, data, and computational power: Cloud technology, the Industrial Internet of Things (IIoT), blockchain, smart grids, edge devices.

Autonomous systems work on specific tasks autonomously without human interaction by leveraging AI algorithms. In manufacturing, autonomous systems can collect information from the surrounding environment, adapt, and make data-driven decisions without the intervention of a human user.

Advanced robotics are systems that combine robots’ hardware sophistication with smart sensors (e.g., ultrasound, light sensors, touch sensors, etc.) which makes them easier to train to perform repetitive and non-repetitive tasks in a manufacturing environment.

Manufacturing analytics: For analysing data collected from manufacturing processes and making data-driven decisions about products, processes, inventories and assets, as well as inferring insights about customers’ needs and market trends. Advanced analytics in predictive maintenance systems could reduce equipment downtime by 50% and increase production by 20% .

Human–machine interaction: Virtual reality (VR) and augmented reality (AR), robotics and automation, autonomous guided vehicles, simulations such as digital twins.

Advanced engineering: Additive manufacturing (such as 3-D printing), renewable energy, nanoparticles.

Cybersecurity solutions aim to protect business data including manufacturing processes, inventory, assets, costs, and client data.

The goal of Industry 4.0 is to improve production, reduce costs, and optimise processes by creating smart manufacturing machines and systems that are connected, automated, and analysed thoroughly.

Impact of industry 4.0 data on HMIs

If we consider these factors we can see that an Industry 4.0 HMI terminal – and its operator – will be subjected to much higher levels of incoming data than a traditional distributed control system (DCS), supervisory control and acquisition system (SCADA), or programmable logic controller (PLC) HMI terminal would be.

Firstly, the amount of equipment of various types and diverse applications that must now be managed is much greater. Secondly, the volume of data generated by each machine may be much higher. For example, a machine that once just fed back real time data about its throughput and current operating conditions may now provide further status data related to, say, motor temperature and vibration to inform predictive maintenance analytics.

HMI and embedded PC manufacturer Advantech sees connectivity and Big Data as key issues for Industry 4.0 HMI design: “In intelligent manufacturing, more control functions are required to process higher data complexity and larger data quantities. HMIs have to be capable of a new generation of communication protocols to ensure the stability and the immediacy of data transmission between PLCs.

“Furthermore, big data analysis is a valued asset when developing a digital transformation strategy, so data acquisition is key to manufacturing and factory operations. HMIs are also required to provide powerful connectivity with PCs or sensors for data transmission through Ethernet or Wi-Fi, but also to support monitoring programs and data collection systems like SCADA.”

How standardisation is helping with Industry 4.0 HMI design

In traditional process or production control systems, HMIs have tended to use a broad spectrum of colours, with unnecessary graphics, visual distractions, and lack of overall situational awareness. Such displays can suffer from inconsistent navigation, presenting data that is difficult to understand, improper depiction of alarms, and a lack of display methodology .

This can lead to poor operating procedures, such as running by the alarms, where an operator is only responding to alarms without understanding the root cause of the alarm conditions.

In other cases, a poorly designed HMI will result in avoidable upsets and increase the likelihood of less than the optimum response to an abnormal situation.

The ISA-101 HMI design standard as a solution: In 2003, the International Society of Automation, or ISA, tasked a group of end-users, operators, and engineers to start working on a standard .

In 2015, twelve years later, they published the ISA-101 HMI Design Standard, titled “Human Machine Interfaces for Process Automation Systems”. This is a set of guidelines, principles, and philosophies for developing graphics on a process HMI. This standard is meant to create a more functional, easy to understand, and information-driven operator interface.

Key ways in which ISA-101 seeks to define a high-performance HMI include :

Proper use of colour: Instead of intense and colourful graphics, the High-Performance HMI is developed in grayscale, with colour intended to be the attention-getter. In a grayscale screen, the use of colour is meant to indicate an abnormal situation very quickly. It has been shown that the new use of colour alone has resulted in a 48% improvement in detecting abnormal situations before alarms occur.

A pump will show as white when it’s running, and dark grey when stopped. It will be shown in medium grey if it is not sending feedback.

Use of information over data: Many HMIs will have dozens or more data points visible on the screen, but nothing for an operator to determine what that data means.

A pressure indicator could read 900 psi, but is that a good thing or a bad thing? By utilising an indicator of normal range with a process variable, the operator can make a quick decision to take action to correct a situation that is trending away from normal.

Trend display graphs can be superimposed onto a tank’s display image. This gives the operator an immediate overview of historical data, allowing them to make any necessary process adjustments before a product runs out of specification or a tank overflows.

A very important concept in High-Performance HMI is keeping the screen simple and uncluttered. A simple depiction of a vessel with a valve and pump is all that is necessary.

The standard also defines a display hierarchy. Creating a hierarchical system of displays gives the operators overall situational awareness, and the ability to drill down to very specific data points when necessary.

The four hierarchy levels are:

Overall situational awareness
More detailed view
Equipment details
Diagnostics

HMI hardware

Currently, a mainstream HMI terminal incudes three core hardware functions; an input device or devices, built-in control electronics or intelligence, and an output device or devices.

Input devices Apart from a keyboard and mouse, input types include touch screen, gesture recognition, voice activation, and buttons.

Touch screens: HMI touch screen type choices depend on the users’ requirements. If the application requires precise and accurate multi-touch capabilities such as zoom, drag, swipe, and pinch touchscreen gestures, projective capacitive (P-CAP) control touch screen types are ideal.

Although capacitive touchscreens are more sensitive and responsive than resistive touchscreens, they require direct contact with a conductive object, such as a finger or a special stylus. In industrial environments with operators needing to wear gloves, resistive touch screen (RTS) types perform better, while being reliable and cost-effective.

Voice activated interfaces: Voice-activated interfaces, also known as voice recognition systems, enable users to interact with HMI systems using spoken commands. These interfaces can be highly advantageous in situations where the user is unable to interact with traditional input devices like keyboards, mice, or touchscreens. Voice-activated interfaces have become more popular in recent years due to advancements in natural language processing and machine learning algorithms. These greatly improved the accuracy and responsiveness of speech recognition systems .

Gesture-based interfaces allow users to interact with HMI systems through physical movements and gestures. These interfaces have gained popularity due to their intuitive nature and potential to provide a more immersive and natural user experience. Gesture recognition systems typically rely on various sensors and cameras to track and interpret user movements, enabling HMI systems to respond accordingly .

Physical buttons were used as input devices long before the term ‘HMI’ was coined – and they are still important today for some situations. They provide great tactile feedback, which is particularly important to those with visual impairments.

Physical buttons are also helpful for eliminating issues encountered when using touchscreen HMIs: namely difficulty distinguishing graphical onscreen changes under sunlight, shifting, and unclear targets, and the need to look at a screen when performing important tasks .

Advantech’s SPC-821-MLA is a 21.5” industrial monitor which accommodates user preferences for physical buttons while providing a touchscreen for more efficient operation, as shown in Fig.1.

Figure 1: Advantech’s SPC-821-MLA 21.5” industrial monitor with pushbuttons

HMI intelligence and computing hardware

Current HMI terminals come as one of two types: smart, or thin client. A smart HMI comprises an integrated assembly with a powerful industrial PC mounted behind the TFT (or possibly LCD) display panel. A thin client has a less powerful industrial PC, or possibly no built-in intelligence – it simply has control electronics for the display panel and touch screen.

HMIs with high computing power are needed for applications processing big data or high resolution images, such as defect inspection applications used in quality control processes for the food and beverage industry, data and image calculation used in electronics manufacturing, and face recognition technology for people working in public places. Advantech, for example, provides premium industrial panel PC solutions, including the fanless touch panel PC TPC-B610 series, and the configurable panel PC PPC-600 series to realise industry 4.0 applications.

On the other hand, if the data computing requirement is lighter, a thin-client terminal HMI with lower power consumption and lower cost is recommended. Additionally, if the panel PC is to serve as a web terminal, Advantech offers a web-browser terminal HMI series with embedded browser and rapid Web App development features. Advantech provides cost-effective industrial thin-client HMI solutions and web browser terminal series to fit different scenarios.

Display screens

Choice of display screen for an HMI terminal depends on the type of application and its physical operating environment.

If it’s on a process floor, it will need IP-rated protection against dust and liquid ingress. It may also need protection against vibration created by nearby heavy machinery. If it is for outdoor use, it will need protection against temperature extremes as well.

Sunlight readability may be an issue in some locations. High brightness screens may also be essential, while contrast ratios also affect readability.

Customisable options, such as pushbuttons, function keys, and bezel colours may also be useful.

If the HMI is for a fairly simple application, such as controlling a single machine, then a smaller, lower cost display taking less space can be used to handle the limited display requirements – see the example in Fig.2.

However, if several machines are involved, or the display is to be used for visualising more sophisticated data such as performance analysis, predictive maintenance, or process optimisation, then larger, higher-resolution displays become necessary. They can present the complex information involved with sufficient clarity and detail.

Figure 2: OMRON NB5Q-TW00B HMI, 3.5" TFT LCD colour display

HMI software

HMI applications range from monitoring single machines to overseeing an entire factory. Accordingly, different types of HMI software are available to handle these diverse requirements:

Supervisory software provides a high-level view of the entire industrial system. It allows operators to monitor and control multiple machines or processes from a central location.

Features include real-time data visualisation, alarms, historical trends, and system-wide management.

Supervisory HMIs are commonly used in large-scale manufacturing plants, utilities, and infrastructure systems.

Machine-level HMIs are dedicated to specific machines or equipment. They offer localised control and monitoring capabilities. These HMIs are typically installed directly on the machine or integrated into its control panel.

Features include machine-specific graphics, alarms, diagnostics, and operator interaction.

Machine-level HMIs are prevalent in individual machines, robots, conveyors, and assembly lines.

Data handling HMIs focus on data analysis, reporting, and visualisation. They collect and process data from various sources (sensors, PLCs, databases) and present it in a meaningful way.

Data handling HMIs are essential for performance analysis, predictive maintenance, and process optimisation.

Some real world examples of advanced HMI technologies

So far, we have discussed various aspects of HMI hardware and software in fairly generic terms – an overview of how HMI manufacturers and users are responding to the Industry 4.0 challenge.

However, we can highlight some of the best of recent innovation by looking at a couple of manufacturer-specific examples, as below.

Schneider electric ecoStruxure™ augmented operator advisor

EcoStruxure™ augmented operator advisor is a powerful tool that combines augmented reality with real-life environments for instant diagnosis and contactless maintenance. It dynamically overlays contextual and local information on a mobile device, creating a seamless fusion of the physical world with virtual objects. Operators can superimpose the current data and virtual objects onto a cabinet, machine, or plant. This innovative solution enhances efficiency and reduces costs.

Figure 3: EcoStruxure Augmented Operator Advisor puts real-time information at your fingertips, whenever and wherever it is needed

Operators can virtually ‘open’ an electrical cabinet and visualise its internal components and layout, or access hidden parts. They can then track the machine’s operating status with various colours on the display. Detection mode allows operators to recognise a scene by tags or 2D images for easy and fast detection of faults in all situations.

They can also free their hands for working on a machine by freezing an image and putting the tablet on a nearby work surface. The app guides operators through procedures by providing step by step instructions on the tablet.

They can inform their maintenance and repair tasks by accessing a wide selection of real-time data from PLCs, documents, images, web pages, notes, labels and data from an SQL Database. PDF files with technical documentation of the equipment, as well as electrical diagrams, images, and videos, are easy to find.

The app can be downloaded onto Android, Windows or iOS tablets, and project languages can be changed on the fly.

The ecoStruxure™ augmented operator advisor is a component of Schneider’s EcoStruxure platform, which lies at the heart of the company’s IoT system architecture. It connects everything in an enterprise from the shop floor to the top floor and collects critical data, from sensors to the cloud. By analysing data to discover meaningful insights, it enables users to base actions on real-time information and business logic. The EcoStruxure Platform is the foundational technology backbone on which Schneider Electric solutions are built and delivered.

EcoStruxure is also supported by hardware products, such as the 7” touch panel display shown in Fig. 4.

Figure 4: Schneider electric HMISTM6400 touch screen, 7", WVGA TFT LCD, 24 Vdc, ecoStruxure operator terminal expert

Siemens winCC

Siemens simatic winCC unified system comprises powerful HMI (human machine interface) software used for visualisation and process control. It allows users to overcome the challenges of digitalisation, and create interactive interfaces for industrial machines and systems. It combines modern and secure web and edge technologies .

With simatic winCC, users can optimise their operator guidance, and are supported in the planning and traceability of production processes. They have flexible access to all data and are provided with efficient tools for analysis and minimising downtime.

Based on web technologies such as HTML5, SVG and JavaScript, WinCC provides good usability, regardless of the device used. WinCC Unified enables authorised operators to access the system via any current web browser – without the need to install separate plug-ins.

From the control panel directly on the machine to complex PC-based solutions: SIMATIC WinCC Unified offers a wide range of options for industry-specific requirements and can be utilised for user-specific applications with its open interfaces. New technologies and open design allows users to easily exchange data from WinCC Unified with other systems.

The technologies used ensure maximum flexibility for WinCC Unified in the choice of the runtime environment. In addition to established Panel- and PC- systems, WinCC Unified can be used in future Industrial Edge environments.

The company also supplies compatible computing and display hardware, including their Simatic HMI Unified Panels. These are a new generation of operator multitouch panels with modern web and edge technologies, for innovative operating concepts and for use even in special environmental conditions, such as in hazardous and hygienic areas or shipbuilding.

Simatic HMI Unified panels are available in both Basic and Comfort versions.

Basic panels are compact, fast in visualisation and have everything necessary to implement price-sensitive solutions.

Simatic HMI unified comfort panels with their capacitive multitouch displays (in sizes 7 to 21.5 inches) offer the freedom and possibilities needed to implement innovative operating concepts. The panels are particularly powerful, allow the installation of apps and simplify scaling thanks to vector-based visualisation.

The unified comfort panels are the latest additions to a wide range of Simatic panels of various sizes, costs, and capabilities, which allows users to choose panels best suited to their application and budget. Fig.5 shows an example of a Simatic panel.

Figure 5: Siemens Simatic KTP700 Basic 7” TFT touch screen, WinCC configurable

Extended reality

Most current production and manufacturing processes are based on traditional technologies, even if they are increasingly embracing digitalisation. Similarly, the HMIs that provide operators with visibility and control of these processes utilise well-established touch screen technologies, although, as we have seen, some now employ Augmented Reality.

However, devices and machines equipped with new Key Technologies like AI and robotics increasingly bring new workflows that include both physical and digital work information. Their use requires constant and intense exchange of information, increasing the complexity of the interaction between human workers and machines.

Traditional human-machine interfaces are insufficient in providing enough context and efficient interaction between humans and advanced machines. This has driven the development and adoption of new interface technologies powered by AI and IoT and pioneered in the creative industries, which have already been applied in a significant scale to manufacturing, and that is rapidly extending to Transport and Logistics, bringing operational efficiency to a level unimagined a decade ago.

This new wave of interface technologies is included under the umbrella of Extended Reality. These technologies enhance human senses, providing additional information, either about the actual world or through simulated worlds for humans to experience, with the objective of monitoring, managing, and making decisions working with advanced machines. Extended Reality includes virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies .

Virtual reality (VR) offers a complete immersive experience in a computer-generated environment. Its use is already widespread in the creative sector and has also been adopted by the retail trade, but VR has little applicability and offers limited benefits to the manufacturing and logistics sectors.

Augmented reality (AR) is a more evolved and useful interface than virtual reality. Unlike virtual reality, which requires the user to inhabit a virtual environment, AR applications superimpose digital information like digital 3D, computer generated graphics and images of the equipment on top of the user’s physical environment. AR thus provides a more natural environment for workers to instruct advanced machines to actuate, often through human physical motion. Its use is particularly efficient when it is necessary to modify the tasks being conducted by the machine, since AR offers an interactive dashboard that avoids costly reprogramming breaks.

Mixed reality (MR) is an emerging technology that combines VR and AR, facilitating working in a real-world environment using some virtual objects; for example, an immersive rendering of the internal components and functioning of a device that the operator is using.

VR, AR and mixed reality powered by AI and IoT can be combined in different proportions, to provide an interface aligned with the specific workflow and tasks being conducted, and the level of autonomy of the machine. Also, proportions are adjusted to facilitate the specific tasks being conducted by the human operator, such as monitoring, scenario testing, reprogramming and task modifications. For example, Mixed Reality interfaces can make proactive, dynamic, and computerised adjustments without written reports, nor direct physical work on the machines.

Onwards to industry 5.0

Industrial manufacturing and process industries must continue to adopt key technologies such as AI and robotics, as well as nanotechnologies, advanced materials, quantum technologies, and others, to remain competitive on an international stage. This evolution is happening, and it is being matched by developments in HMI technology.

Yet new horizons are appearing as we meet the challenges of the earlier ones. One development is Industry 5.0, which – among other things – reintroduces people to the automation loop, allowing people and robots to work much more closely together. In a symbiotic relationship, humans will be able to work alongside a new generation of collaborative robots (cobots), adding value to products.

Production lines can become increasingly smart, with humans being able to oversee much higher levels of product customisation. That’s an exciting thought in areas as diverse as electronic devices and jewellery, where added touches to product finishes can result in higher consumer appeal . And it will be interesting see how HMIs contribute to more symbiotic relationship between machines and human minds.