The logging of historical process data in industrial automation systems has been widely mandated by regulatory bodies for a couple of decades. Its purpose serves to prove a business’ operating compliance with local codes and regulations and for product quality control. Alarm information, however, tends to have a much shorter “shelf life” and often doesn’t even get archived. The fact is that when categorized properly and by using predictive modelling such as clustering analysis, historical alarm data can reveal very useful information about existing and potential issues with the processes.

Our Client is a successful business selling product in a highly competitive segment of the retail market space. Their processes are straightforward, they run fairly lean with a good mix of automation and support staff to run day-to-day operations. Although they have the visualization technology into their processes to address plant issues transactionally, they realized that they couldn't get a concrete understanding of the causes for their production problems and that’s when they decided to do something about it. Where to turn to?

Understanding that the way that they were correcting problems were temporary fixes but that they had data to apply advanced unsupervised machine learning algorithms, they accepted to take a leap of faith to try that different approach. As the intent was to understand the yet unlabeled data, clustering was the modelling method chosen. Clustering is a form of unsupervised Machine Learning that presupposes that very little is known about the different classes of the data.

The plant alarms were clustered with two (2) main goals: The first is prediction – separating issues that require different solutions. The second is to provide a basis for analysis - identify the causes of the different types of issues. It is these two goals we had in mind when we made recommendations to this Client.

Below is an example of the method used to tease out the issues. As this type of analysis can become computationally intensive with large datasets (i.e. > 500 points), Cloud services such AWS or IBM SPSS should be considered.

Methodology Used:

Problem Statement: Systemic Causes of production problems.

Measure: Categorize alarms, obtain a representative subset of process alarms logged over a representative time period.

Analyze: Start with a higher-level view of possible quantity of representative clusters (Dendogram). Perform deeper

learning clustering analysis, such as PCA, and identify possible associations.

Implement: Identify issues and causes, consult with plant SMEs as required to validate.

Control: Design and Implement relevant Engineering Controls to address issues.

The Client benefited from this analysis as it revealed associations between data objects which were not apparent previously. An action plan was developed. Among other process-related improvements, the need to add a network-based firewall was identified to prevent suspected disruptive network activity originating outside the automation system. Environment related issues were also identified.

References:

[1] Darveau, P. Prognostics and Availability for Industrial Equipment Using High Performance Computing (HPC) and AI Technology. Preprints 2021, 2021090068 .

[2] Darveau, P., 1993. C programming in programmable logic controllers.

[3] Darveau, P., 2015. Wearable Air Quality Monitor. U.S. Patent Application 14/162,897.

Summary

Successful safety retrofit projects require engineering processes flexible enough to incorporate changes to promote the flawlessness of their execution. In these types of projects, timing is everything. The cycle times between design and contractor selection continues to get compressed as budgets only get approved based on cash flows orchestrated by client Finance departments. Lean Six Sigma (LSS) methodology provides a useful roadmap leading to successful execution of these types of projects.

Why are these projects different?

Retrofits are an important part of the engineering marketplace. As infrastructure in the plants ages beyond 20 years, important upgrades are required to keep the critical infrastructure in tip top shape for the next economic upswing, maintainable and capable of meeting the latest safety practices as regulations mandate. The nature of these projects also makes them the most challenging technically. These projects are different operationally as they almost always require a major shutdown to be deployed making timing extremely crucial when taking them on. They also involve a regulating body. Missing the schedule could mean delaying the install of the retrofit 18-24 months and, in worst case scenarios, cause unplanned downtime for clients due to obsolescence of components! Firms in this business have to be lean and nimble even with established engineering control processes in place, hence the suitability for Lean Six Sigma methodology.
Enter value mapping and waste elimination

Besides the list of documentation to be produced in the design of a safety retrofit, there are two (2) very important pieces of information that must be determined with certainty:

1) The Code(s) to follow;
2) The Certifier’s expectations for approving your design before Issuing for Construction (IFC).

Determining the applicable Codes comes with the territory and should be known by the engineering consultant right from the Proposal stage. Planning for Certifier expectations with the goal of obtaining approval at the IFC and as-built stages is key to a successful project. This is where mapping your value stream and performing cause-and-effect analysis comes in to reduce design/approval cycle time and waste (especially wasted time) in the engineering process. If the contents of the following chart look familiar, please continue to read on.

Where do we go from here?

Use the Six Sigma DMAIC method to Define, Measure, Analyze, Implement & Control. Our roadmap (a less generic DMAIC, if you like) boils down to this:

Document it. Don’t defer fixing a cycle time challenge. Plan to fix it;
Mitigate risk by developing a project execution plan (PEP) and maintain a schedule;
Ask for a value map and perform cause-and-effect to determine causes of wasted time and changes to process. Seek some help to do this;
Inform all leads of solution(s). Inform the teams. Be specific and;
Consider integration of solutions into relevant IT processes.

What was learned?

Rules of Credit (I.e. 30%, 60%, 90% stages) applied to engineering and construction deliverables based on properly established progress criteria improves execution time by up to 15%. Implementing procedures using LSS methodology can be an effective way of putting procedures in place to consistently benefit from this improvement.

Next step

Driving consistency will best be served by integrating the improvements into the IT applications that support the Project Management processes. This should follow your business’ documented IT strategy and framework.

Thanks,

Peter

 The Role of Project Managers and Subordinates in different project structures.

Is your Project a hierarchical, a flattened or a network structure ? It turns out that there may no quick answer to this question. Projects, like organizations, are dynamic entities that learn, and hence change, throughout their execution. A learning Project Manager recognizes this and adapts the leadership role according to the behaviour, trust and confidence within the team.

A few years ago, I worked on a Project as project manager where a changing situation occurred in the structure not by managerial intervention but by team dynamics. What I learned was that my involvement needed to change from that of a “decision-maker” to a “coach” being available as required to situational leaders who bubbled up in the Project team. The following figure describes the evolution pictorially.

   Hierarchical                 Flattened                     Networked

What I noted throughout this process, was that more decision-making was required of the Project Manager at the early stage (Hierarchical). As the project moved forward and transitioned, the teams ended up making their own decisions with the support of situational leaders (Networked). Since my role became more “hands-off”, I found that I could let the project run autonomously and focus my time and improve my efficiency on other important matters on my plate.

If you manage projects, you may be asking yourself how do project updates get communicated when the Project turns into a Networking structure. As the team is more self-disciplined in this scenario, an easily accessible huddle board containing the deliverables milestone dates and budget may be considered. By walking up to the board frequently throughout the day, the team stays the course. A huddle board also makes schedule discussions simpler than setting up weekly dedicated meetings for this purpose.

The experience I described above was not smooth nor was it precisely timed nor calculated, but it ended up being successful nevertheless. After seeking some coaching and support, I decided to let the team run in a networked fashion only when the message was loud and clear from the situational leaders that they would own the Project. This experience was yet another item to keep in mind for my next Projects.

The lesson learned is that teams that can self-manage are a significant benefit to any organization. Finding a way to foster this type of behaviour, I believe, is Golden to any Project and its organization.

Thanks,

Peter Darveau

Executive Summary

Control Systems (CS) have come a long way over the past 25 years. In the electrical age that we are living in, these devices (some as big as a bread box, others as small as a credit card) have flourished in all aspects of industry from plant floor machine controls to semi-sophisticated data acquisition systems, from motor starters to “Smart” MCCs. They are appearing in instrumentation and sensing and, with the advent of Industry 4.0 (Internet of Things in North America) demanding increased interconnectivity of devices, they will appear in even more conspicuous applications such as Air and Water Filtration, Purification and Decontamination systems, for example, which are subject to national, regional, municipal or industry-specific regulations.

Why it matters

Cyber Security at the CS level (Level 1 per ISA-95) has been relatively tame thus far; the Stuxnet malware attack being probably the most widely remembered. The proprietary and closed nature of legacy Operating Systems explain the minimal presence of this issue. Newer controls, however, are based on more open, less controlled platforms, such as Linux. The increased sophistication of cyber attacks is bound to increase the risk of threats as they move from Advanced Persistent Treats (APT) to memory-resident and fileless malware. As a matter of fact, vulnerabilities of the most well-known CS vendors are now being published in an attempt to address this increasing risk to critical infrastructure. According to Kapersky Security Researchers, cross-site scripting (XSS), buffer overflows and compromising of credentials will account for 20% of attacks on ICS over the coming years.

Where to Start

Mitigating these risks starts with a Cyber Security Plan. The Plan establishes a means to achieve a high assurance that Electronic and Programmable Electronic systems and communication networks associated with the following functions are adequately protected against cyber attacks:

Safety-related and important-to safety functions;
Security functions; and
Support systems and equipment which if compromised, would adversely impact safety, security, or emergency preparedness functions.

Some Recommended Best Practices:

Develop a thorough Cyber Security Plan. Many organizations will find it efficient to have an independent automation consultant to do it for them.
Questioning your vendors. Many vendors have been slow to mitigate risks within their platforms. If you see weaknesses, identify them and ask your vendors about implementing effective solutions. Become informed about their existing documented vulnerabilities and recommended mitigating actions.
Planning to migrate to newer technology and budget it as a necessary cost of business if your CS network relies on older Microsoft or proprietary operating systems. While many CS are designed with a systems life ranging from 15 to 20 years, older systems may be able to accommodate the rapidly changing cyber environment we face today.
Hire a reputable firm to review software update deliverables prior to installation in the production network. Your preferred controls integrator or consultant is the best place to start. Consider having static memory analysis performed to detect Rootkits. Assume this gap will be breached and plan your response accordingly.

Thanks,

Peter Darveau

Executive Summary

An employer's due diligence as part Occupational Health and Safety (OHS) legislation means that all reasonable precautions, under the particular circumstances shall be taken to prevent injuries or accidents in the workplace. When working in environments where airborne particles are present or where certain gases can attain toxic levels, monitoring is often required. However, traditional methods for monitoring air aren't always reliable and, if an incident occurs, the question industry should be ready to answer is: "Was there due diligence ?"

Common Practices

Common practices involve a number of strategically located stationary panels that monitor specific gases in their respective location. The problem with this approach is that gases move with air flow which is affected in an unpredictable pattern from day-to-day operations of a plant. To put this into context, toxic concentrations can exist in "pockets" that float around in a given area and go undetected by the stationary panel or station. The personnel in the affected area won't always be aware of the increased exposure while the monitoring equipment will signal back to the Master Control Room (MCR) normal environmental conditions.

Evolving Practices

In an article I published in April 2015, Hexagon Engineering's Wearable Digital Air Quality Monitor was introduced as part of a Study group. Now, we bring the practice a step closer to an industrial application. The device is light weight, fits in a pocket or clips on a belt. Visual indication is available on the device itself or optionally on a smartphone via a wireless communications link. At the site seen at this link Site Test, monitoring would be for levels of Carbon Monoxide (CO) and PM2.5. Exposure to both measurements can be alarmed in real time for the location where the employee is situated, filling the gaps (and potentially all requirements) by the stationary panel shown in the picture.

Next Stage

Looking forward, the next stage will involve the application of this technology for industrial safety applications using predictive analytics and cause analysis as part of Industry 4.0 deployments. In the longer run, these are the IoT initiatives that will allow industry to better conduct due diligence in their OHS programs.

If you liked what you read, please feel free to share this article with others you know via LinkedIn, Twitter, Google+ or Facebook. It’s good to share.

by Peter Darveau

For more about Hexagon Technology Inc., visit www.hexagontechinc.com .

Controllers in safety applications form risk mitigation systems usually known as Safety Instrumented Systems (SIS). The development of these systems over the past 10 years has seen an increase in performance, reduce their footprint by more than half and allow integration with the higher-level Process and Operation digital systems. The latter has resulted in more awareness that networks in critical control applications don't share the same requirements as those of a corporate network. The design of these safety systems, taking into consideration cyber-security and network performance, is taking a new turn that Software Defined Networking (SDN) is driving. Over the past deployments of the Hexagon Technology Inc. (HTI) Safety Control Unit (SCU) in SIS applications, discussions with peers and customers concerning the impact SDN will have on the future development and design of the SCU have taken place. To understand those effects, let's touch on the following key points.

First, a brief introduction of Software Defined Networking (SDN) and reasons clients are considering this approach. SDN is a means by which all data flow control decisions in an Ethernet-based network are centralized in a single Controller. Connected to the Controller are simple devices that forward data according to pre-defined instructions over the same CAT5 or CAT6 cables used in today's networks. These simple devices replace the routers and switches distributed throughout a typical network that we are all familiar with today. Because of this centralized nature, SDN provides the means to comply with ISA84-TR84.00.09 (Security Countermeasures for SIS) in the best possible way and fulfills the critical infrastructure requirements in reliability, deny-by-default security, latency guarantees, and deterministic transport capabilities.

Second, it is important to note that the SCU is a SIL3 rated unit with the sole purpose of performing advanced diagnostics and, in certain rare cases, triggering a safety function. The dual or triple voting controllers and communication hardware of the SIS to the Process Controller (DCS), on the other hand, are based on standard off-the-shelf components from global automation suppliers. Since the SCU is contained within the SIS and is not intended to connect to the outside process controllers, it is designed to be a component that is part of a certified system. This means that once the certification is obtained no change to the configuration should occur without the proper risk and validation (HAZOP) and SIL Determination tasks first taking place. In short, the SCU should be decoupled from a control network and shouldn't be part of a Software Defined Network.

However, proof of concepts using SDN to enhance network reliability to never seen before levels are in the works in the Energy industry using predictive software tools to diagnose and prevent communication failures by failing over to alternate paths. Favourable outcomes in this area are expected in 2016 and will lead to new ways of achieving higher levels of safety capabilities with lower cost hardware. The SCU is uniquely designed to incorporate new network functionality enhancements driven by SDN due to its supplier-neutral platform architecture.

On a last note, although SDN provides interesting opportunities to Hexagon Technology's SCU, proven key encryption algorithms and certificate management are recommended for ensuring proper network security to the overall process control / SIS system.

If you liked what you read, please feel free to share this article with others you know via LinkedIn, Twitter, Google+ or Facebook. It’s good to share.

by Peter Darveau

For more about Hexagon Technology Inc., visit www.hexagontechinc.com .