The Why and Hows of IIoT Implementation: The View from a Regulated Industry
Łukasz Paciorkowski starts by asking two critical questions: first, why should we implement IIoT solutions, and second, how should we do so? He convincingly demonstrates why companies should invest in the Industrial Internet by offering a real-world example taken from an ongoing project in the biopharma industry, one of the world’s most restricted fields.
Barriers to IoT Adoption
Almost every article about the Internet of Things (IoT) starts with predictions of how many connected devices there will be beyond 2020. Those statistics try to catch our attention and make sure we understand (and agree with the assumptions about) how big the impact of IoT will be on consumer electronics, various industries, and the global economy in general. Proponents rave about the connected toasters we will be able to operate remotely using any smartphone.
But the reality is that organizations are often not able to clearly answer the simplest questions regarding the business case for IoT implementation. After the initial excitement — often fueled by consulting and market research companies — comes the hesitation and doubt: is IoT really such a cool and useful concept? Figure 1 shows the three main barriers to IoT adoption.
Why Should We Implement IoT?
The first barrier to IoT adoption is the question “Why?” Why should producers release connected products in the first place? The famous “connected toaster” sounds great until you start to analyze implementation and maintenance costs versus potential benefits. Creating a comprehensive business case for IoT is not a straightforward thing. Not only are connected products a fairly new and unexplored concept without well-defined and proven revenue models, but there are additional factors such as technology-related fees and new ways of selling products. All of these factors impact time and effort estimations and financial projections.
How Do We Implement IoT?
The second barrier to IoT adoption is the question “How?” Traditional product companies rarely have the knowledge and capabilities to envision, design, implement, and maintain the complex IT systems that provide advanced communications and data processing capabilities. Many manufacturers face an almost existential question: are we still a product company or have we already become a software company? How do we bring new talents to the organization? How should we manage projects that are so different in nature from the ones we did in the past?
How Do We Ensure IoT Security?
The third barrier is security, the most often-used argument against implementing IoT. A justified fear of “being hacked” makes business stakeholders think twice before jumping on the connected products bandwagon. Recent publications showing just how unsecured connected products can be serve to discourage even the most enthusiastic innovators. Consider the damage that could be done to an established product brand due to an IT hack. Is IoT worth it?
In this article, I will briefly discuss why companies should invest in the Industrial IoT (IIoT), offering a real-world example taken from an ongoing project in a global life science–biopharma organization. Then I will focus in greater detail on the second barrier: how to implement IIoT and address the challenges related to the implementation project. As we will see, the “how” does not relate solely to the technical aspects. In fact, technology is just one item on the long list of impediments connected to IIoT adoption. Organizational, cultural, and process-related challenges consume the greater portion of the time, money, and resources needed for IIoT-related efforts.
A Real-World Why: IIoT’s Value Proposition for Biopharma
Let’s begin by considering a potential business case that could justify investment in a connected products initiative. The example we will look at comes from biopharma, which is considered one of the world’s most regulated and controlled industries.
In biopharma, a bioprocess is used to produce a wide range of drugs and vaccines leveraging living cells. The process itself consists of multiple steps — from upstream bioprocessing, where cells are grown and cultivated, to downstream processing, where the product (cells mass) is purified, concentrated, and polished. The entire process is enabled by equipment that provides the environment for the bioprocessing. This equipment consists of many different Unit Operations (Unit Ops or devices) used to perform specific steps in the bioprocess. Unit Ops available on the market fall into one of the three categories: manual, semiautomatic, or automatic. As we might expect, IIoT initiatives aim to automate the end-to-end bioprocess, eliminating manual actions and assuring higher performance and quality of the entire production process.
Some of the advantages IIoT could provide for the biopharma industry include:
- A higher level of automation. As a bioprocess consists of multiple steps performed on multiple devices using a range of consumables, automated orchestration of data collection and data processing is a highly desired feature. In more detail, there are four main issues to be resolved:
- Integration. Without automatic integration, activities within the bioprocess have to be performed by the human operator. Obviously, such a requirement increases the cost of labor and creates more opportunities for human error.
- Data acquisition. Data acquisition and, specifically, a track-and-trace feature are critical from a regulatory and compliance standpoint. Each consumable used within the process needs to be properly evidenced and recorded. Automating this part of the process using sensors, wireless tags, and other techniques provides a great improvement opportunity.
- Data integrity. With more data being collected and analyzed, data integrity is growing in importance. Ensuring that data was not tampered with, altered, or deleted is critical for security, safety, and compliance. Especially in drug- and food-related industries, implementation of regulations such as 21 CFR Part 11 or EudraLex is a must. IIoT with strong data encryption capabilities can satisfy “hard” system requirements, while the “soft” ones can be addressed through proper process and standard operating procedure (SOP) definitions.
- Reporting. Batch reporting, which relies on documenting all data related to the manufacturing process (order and the details of every step) and underlying conditions (data from sensors like temperature or pH value), is critical in the biopharma industry. Without the proper collection of the data, the final product cannot be released to the market, and the batch will be considered a waste (or loss, in business terms). End-to-end reporting is a perfect example of where IIoT capabilities can bring significant benefits, savings, and optimization.
- Integration. Without automatic integration, activities within the bioprocess have to be performed by the human operator. Obviously, such a requirement increases the cost of labor and creates more opportunities for human error.
- New capabilities and services driven by data. Data collection is nothing new in software automation. Nevertheless, historically, data in automation was used in a limited manner (e.g., directly supporting the manufacturing process). With the advent of advanced analytics, machine learning, and deep learning, new opportunities emerge. Process-related improvements discovered through data mining or predictive maintenance are examples of putting data to good use. Additionally, data integrators (vendors or service providers) can offer additional services based on insight-rich, cross-customer data lakes.
- Lower setup and operational costs. In biopharma, as in some other industries, a new breed of startups are emerging: young firms with big budgets that want to set up their own bioprocess production. Without deep expertise or know-how, such organizations rely heavily on the external expertise and out-of-the-box functionality delivered with the automation platform. A high level of standardization and a comprehensive set of functions drive the design of modern, future-proof IIoT platforms. In addition, new and established market players are constantly searching for ways to lower the risk and cost related to compliance. Standardization of data formats (e.g., B2MML), protocols (e.g., OPC UA), and integration capabilities provides an optimization and cost-reduction opportunity for both startups and existing companies. Surprisingly, for a traditional, regulated, and closed industry like biopharma, usage of cloud is being discussed as well. It is seen as a way to shorten the release and time-to-market cycles, while at the same time lowering CAPEX investments.
These are some of the most obvious benefits the IIoT could offer the biopharma industry. Further analysis of business models, processes, and market segmentation provides many more arguments for IIoT implementation. Assuming we are already convinced that connected products are the way forward, let’s now look at some challenges related to the IIoT implementation process.
How #1: Bridging the OT-IT Gap
For years, operational technology (OT) and information technology (IT) have lived in their own worlds. OT concentrated on operational excellence, safety, and reliability, whereas IT focused on rapid innovation, new technologies adoption, and continuous delivery. Indeed, those worlds grew so far apart that it takes quite a bit of time before OT and IT engineers can have a meaningful conversation. Technologies, definitions, wording, and even their ways of working and thinking are so different that it is almost impossible to imagine how the two could cooperate. And yet IIoT is bringing these two worlds together.
For manufacturing companies, connected devices, sensors, and automation are nothing new. Over the past 40 years, industrial software platforms have steadily grown, bringing reliable automation to the manufacturing sector to help car manufacturers, oil companies, utilities, and many more organizations optimize and automate their operations. Including sensors in every moving component is bread and butter for seasoned automation professionals. So what is different about the IIoT?
One of the most significant changes in current industrial automation trends is that, until recently, OT lived in its own closed ecosystem. Starting from specific software like DCS or SCADA, through costly hardware like PLCs and industrial sensors, up to dedicated network infrastructure, OT was connected but isolated. Cyber threats so common to the IT world were mitigated by cutting any external access to the industrial automation systems. Compliance and regulatory requirements along with international standards made OT a very stable environment. In many industries, upgrading an operating system with newly released patches is a totally novel concept. Once validated and qualified, a piece of equipment could work without a single update for years! Confronting this reality with, let’s say, smartphone upgrade frequency shows just how different IT and OT lifecycle management is.
Yet as early as 2010 the industrial world was clearly shown that the separation of IT and OT was not exactly working. That’s when the Stuxnet virus successfully penetrated isolated industrial computer systems and targeted Iran’s nuclear power plants, causing irreversible damage. This was one of the most sophisticated cyber attacks the world has ever witnessed.
Despite growing security concerns, business stakeholders across industries were put under greater pressure to deliver innovative solutions faster, while at the same time lowering operational costs. As a result, industrial automation practice has been forced to move from a connected but isolated environment to the globally connected ecosystem. With this new reality, manufacturers must embrace new opportunities but also prepare for new threats and challenges coming from the traditional IT world. This is where the aforementioned IT/OT convergence is taking place.
How #2: Rewiring the Organization’s Collective Brain
Although a fair amount of OT innovation was introduced over the years, it was nowhere close to what was happening in the pure IT world. Today, however, the rate of innovation is rapidly increasing, even in the most regulated and conservative industries. This is caused partly by the advancements in technology and partly by geoeconomic and globalization trends. Quality comes with an ever-decreasing price tag. Emerging market players can successfully compete with well-established Western corporations. The competitive ecosystem is changing — with IIoT-based capabilities being at the center of this change. Doing “things” the same way is not enough to retain market share.
In our biopharma industry example, the traditional business model relies on selling a product: a standalone, often disconnected, and self-sufficient device. Software automation was a necessary addition and was treated as an integral part of the product. With IIoT platforms coming into the picture, new data-driven services influence how we look at the final product. Suddenly the software and data platforms are as important and valuable as the hardware product itself. A logical step for vendors is to stop talking about products and move toward services. This transformation is very similar to what happened in the IT hardware market after the concept of cloud was introduced and the “as a service” revolution took off. It can bring a lot of benefits, but if not managed properly, it can kill the organization.
Going from being a product to a service company is a truly transformational experience. This shift does not impact only one group or one department in the organization. It requires rethinking the entire organization, starting from the foundation of business and revenue models, through go-to-market strategies, and on up to organizational structure, processes, skills, and capabilities. This cannot happen without the support, understanding, and dedication of people on all levels in the organization: the board of directors and C-suite, middle management, and the operational staff. This is also the moment where the most pushback can be seen from the participants.
Changing the way of working is never easy, especially in such a conservative and traditional industry as biopharma. Continuous education, discussions, and leadership involvement are necessary to manage the transition to a new reality. Not everybody will agree, not everybody will get it, but eventually rewiring the organization’s collective brain has to happen. Although regulated industries tend to be very stable, as the barriers to entry for new market players are high, even there exponential technologies such as IoT are disrupting the status quo. It is always better to be the disruptor than to be disrupted.
How #3: Ending the Clash of the OT and IT Titans (Process, Security, Organizational Transformation)
Good manufacturing practices (GMP) are well-established, globally recognized, and widely applied guidelines for food and drug production processes. Good automated manufacturing practices (GAMP) guide the software automation domain in the pharmaceutical industry. At the core of both GMP and GAMP is the quality assurance process. The behavior of the manufacturing system has to be unconditionally predictable and clearly defined. Each deviation from the standard process has to be recorded, approved, and managed by a dedicated procedure. Quality is embedded on every level: from the equipment design and production, up to the manufacturing line assembly, configuration, and operation, and ending at the device’s decommissioning.
OT engineers and specialists live and breathe those regulations and practices every day. The waterfall development/delivery method is thus a natural choice for them. The V-model for software development — an extension of the waterfall process — encourages up-front design and long delivery cycles. This method of project delivery worked for years. Why change it?
On the other side, there is a “new” IT delivery model characterized by flexible planning, fast delivery iterations, and a fail-often/fail-fast mentality. Agile practitioners do not understand five-year-long release plans. What matters is the current sprint (or iteration). Young IT professionals are using the newest (but not always the most reliable) technologies available on the market. They ask:
- Why use 30-year-old technology if Facebook just released a new front-end library?
- What do you mean “It has to be Windows”? Don’t they use Linux in the automation world?
- Microservices are the way forward. But how do we install Docker and a Kubernetes cluster on a single embedded PC? Does that makes sense?
- And finally, what is wrong with this quality assurance? Why do they check everything that we do and deliver? Do we really have to document everything?
And on it goes …
The differences in the IT and OT approaches to system design are so profound that it takes a lot of energy to bring those two worlds together. Yet IIoT requires this convergence (see Figure 2). There has to be alignment on the technology stack, data models, integration method, reliability targets, and security policies. Both the IT and OT worlds have their own points of view, experience, and best practices. Willingness to listen to the “other side” is a must. An IT engineer will not succeed in the automation world without support from an experienced OT colleague. Similarly, an automation engineer will have a hard time navigating in the new IT environment. They both have to work together and be open to learning from each other — and there is a lot to learn.
Safety, Security, and Speed of Delivery
Two very important differences between OT and IT concern reliability and security. Traditionally, if a system fails in the customer-centric IT world, the worst-case scenario is that a user will complain about the poor experience. If a system fails in the OT world, somebody might die. It is important for IT architects and engineers to understand this significant difference and ensure that the reliability requirements are properly reflected in the system design.
Security by design is a natural thing for IT. Modern IT platforms are designed to be exposed and to be resilient in the face of any type of cyber threat. In contrast, manufacturing systems need to be safe, not necessarily secured. It is hard to ensure the security of a platform that is updated once every few years. The standard approach to security in the OT world is system isolation: limiting access to the platform, cutting all connections, and controlling user interactions with the processes and SOPs. But IIoT requires connectivity on multiple levels. OT can learn a lot from IT when it comes to introducing advanced threat modeling, implementing modern security controls, and practicing appropriate lifecycle management.
There are many more differences between OT and IT that are encompassed in their respective ways of looking at system design and implementation — namely, waterfall (V-model) versus Agile (iterative model). How can system implementers stay flexible while at the same time respecting GMP/GAMP principles? This is where a huge amount of effort is required.
One answer would be to move from one V-model to a model made up of multiple small Vs (see Figure 3). Instead of designing and planning an entire system in a single go, let’s follow the Agile, iterative approach, while also respecting all of the quality assurance requirements. That way we end up with a delivery process in which the complete V-model is executed in each sprint. All quality artifacts (tests, risks assessments, etc.) are produced together with the code during the sprint execution. The quality department becomes an integral part of the Agile development team. This is not easy, and it requires a lot of discipline, but it is doable.
Impact on the Organization
With all the changes described above, it is natural that many other parts of the company will have to adjust as well. The organizational structure should transform in preparation for maintaining and delivering new types of services. The IT organization needs to prepare for new kinds of systems and platforms being introduced to the corporate ecosystem. Go-to-market and sales teams must educate themselves in new offerings, services, and business models. Finally, the entire organization has to continue innovating, since in the hyperconnected world of IIoT, things will change faster than they ever did before.
The organization’s transformation will not happen immediately. Acquiring necessary skills (or developing them in-house) will take time. Based on the experience of the biopharma industry, it makes a lot of sense to use external consultants, experts, and coaches. This approach significantly shortens the learning curve for the organization and the people in it. Creating multiple lightweight proofs of concept that confirm (or don’t confirm) vision and design assumptions will lower the risk for the overall IIoT program. Using an Agile delivery method will help the organization adjust its course and introduce improvements early in the development process. Cooperating across multiple departments is critical for success.
IIoT is an enabler. It provides plumbing for data-driven capabilities and services. Connected products are an interesting concept, but the real value is in the data produced by the devices. That is why it is so important to have a clear vision and strategy on what data to collect and for what purpose(s). Some concepts worth exploring together with IIoT are:
- Advanced analytics with predictive modeling. Analytics and deep data mining can be used both for optimization purposes as well as for exploring and discovery. Applying techniques from the artificial intelligence (AI) domain on IIoT data sets seems to have great potential.
- Blockchain. When data generated by the devices represents a value that can be traded or exchanged, blockchain can be helpful. Blockchain is already used in the energy industry, where electricity trading is governed by distributed ledger. Blockchain can also be used for managing a digital trail attached to a physical object. In the biopharma industry, the products or consumables genealogy can be managed using distributed ledger.
- Self-managing control platforms. With sufficient sensor data, advanced real-time analytics, and AI, it is possible to create a self-managing, self-adjusting, and self-optimizing control system. Such a platform would be able to detect and correct deviations in the process, adjust parameters accordingly, and properly record the actions taken. In laboratories, self-managing control platforms could take care of performing experiments and refining process parameters continuously to find the optimal setup. With enough data from IIoT instead of physical experimentation, predictive process simulation could be used.
Researchers and system designers are constantly exploring these and many other ideas. The IIoT represents a big opportunity for all industries, even regulated ones. However, it also requires new system designs, new tools, new processes, and — above all — a willingness to change. In the coming years, we will see more and more connected products with amazing capabilities released to the market. Pressure to reduce costs will push manufacturers to search for additional savings related to optimization and automation. In the center of this change is the IIoT with its interconnected sensing-and-reacting systems and platforms. It is important not to miss this opportunity and to start exploring the IIoT as soon as possible. You can be sure that your competition is already doing so.