Providing an enterprise-wide data store has been one aim of the enterprise data warehouse since the 1990s. One of the key lessons learned was that resolving issues of meaning and context — metadata — was central to any successful implementation. The challenges remain: very few data warehouse teams have claimed anywhere near complete success. It is also interesting to note that these issues have, finally, been recognized by data lake proponents. Tools offering big data governance, data wrangling, and similar function have begun to emerge over the last year or so. Unfortunately, once again, the tools precede an understanding of the true extent of the problem: how to traverse from data and information to knowledge and finally meaning and vice versa?

Cutter Fellow Ward Cunningham, who coined the technical debt metaphor,[1] observed that when the develop­ment process leads to new learning, re-executing the project — or parts of the project — could lead to a better result. For this reason, among others, newly developed operational software assets can contain, embody, or depend upon artifacts that, in hindsight, the developers recognize could be removed altogether, or could be replaced by more elegant, effective, or appropriate forms that can enhance maintainability and extensibility.

Technology backlash is as old as technological innovation. It is inevitable that people will grouse about new technologies and adopt them with varying degrees of acceptance. Yet, with one caveat, the cool stuff will take hold and prevail on the basis of its functionality and actual worth to people. The caveat is that this will happen only if these products do not give people some absurd reason to do a double-take and say, "What? You didn't tell me this amazing product" -- and here, take your pick -- "uses triangulation to share my location with perverts," "shares my aimless meandering around department store aisles with marketers," "leaves my television camera running," or "records my child babbling away to a beloved toy."

Agile as a concept and as a "method" is invaluable. But have you ever wondered how Agile itself was developed? Did the signatories to the Agile Manifesto use an "Agile approach" to arrive at the manifesto? The science of methods has this age-old conundrum: which method was used to develop a method? And how was that method validated? I share my thoughts with you in this Advisor.

In a recent Cutter IT Journal (CITJ) article ("Leveraging EA to Incorporate Emerging Technology Trends for Digital Transformation"), Cutter Senior Consultant and esteemed colleague Bhuvan Unhelkar and I presented our practical experience in leveraging EA to incorporate emerging technology trends for digital transformation. In the article, we shared the following steps that EA as a discipline can use in facilitating digital transformation:

IT governance defines an organization’s structure, processes, and controls used to oversee the business alignment, planning, and budgeting for the technologies the organization utilizes. The composition of the IT governance group varies greatly from organization to organization. It can depend on the type of business, the size of the organization, the complexity of the technologies used, and other factors. IT governance can range from simply the head of the IT department presenting plans and budgets to the firm’s management for approval to a mature technology committee composed of business executives, firm management, and the CIO, who together oversee and approve the approach to technology acquisition and operations.

Digital transformation continues to change the financial sector. The increasing use of smartphones and tablets has changed customers’ behaviors and fueled adoption of mobile banking. Wearables capable of storing and processing data allow us to integrate better with the technology and incorporate electronics into every domain of our lives. The question arises whether the wearables can follow that success and become a new disruption for the financial sector. And if so, will they replace or supplement the mobile devices that are currently in use?

Organizations have worried about how to protect sensitive data in big data platforms since they were first proposed for enterprise use. This is because big data environments have typically lacked the extensive security features available with the more traditional relational data warehouses that companies have become accustomed to.

In a well-functioning team, collaboration and knowledge transfer are simply byproducts of the team’s work. The fact is that embracing a business process management system (BPMS) results in a certain amount of culture shock for the uninitiated; the learning curve from a technical and business process modeling standpoint is considerable. All of these elements should be discussed at this review meeting.

As part of our shift to cloud and open source platforms, we chose to introduce a number of culture changes. Early in the process, we decided to make the cloud migration a rallying cry for the technology organization. Specifically, we challenged senior technology staff regarding the fundamentals of what our systems did and how well they served the business. This resulted in key changes in the way we addressed the fundamentals of our multi-petabyte, big data problem. In this process, new high-potential technology leaders were identified and elevated in the organization. The hiring and staffing effort that accompanied this effort also provided an opportunity to further reshape the technology profile of the company.

The architecture of the Web is elegant, and it delivers value. Is it possible to have complex, convoluted architectures that are inelegant, but that enable value? Maybe. But my intuition and experience suggest otherwise. I think that architecture needs to be as close to invisible as possible to be valuable. Architecture needs to be the invisible hand that guides the enterprise and the people within in and around it.

Many organizations are opting to go with a commercial IoT platform provider in order to benefit from the convenience that using such offerings provide. The biggest reason is to take advantage of the flexibility, dynamic scalability, and performance offered by cloud-based architectures and services, including the capabilities for publishing APIs and Web services to facilitate the exchange and integration of machine data with enterprise data and to develop analytic capabilities that can be easily consumed by enterprise applications.

In Part I of this series, I explored the data lake and other metaphors for data storage in use today. Leaving technology platforms aside (which I strongly urge you to do, at least for now), understanding what a data lake could or should be starts, unsurprisingly, with knowing what the business needs from it and that it cannot get anywhere else. That understanding comes from a conceptual architecture, which therefore must have a much broader scope than its central topic. A conceptual architecture is a picture that forms the basis for conversation, understanding, and agreement between business and IT. It doesn't have enough detail for IT to build it. It must be simple enough for business people to take it in and understand what's going on.

No enterprise big or small should fail to think about what digital transformation can do for — or to — them. Within commonsense economic limits, current technologies should be exploited. Further, it's important to envision potential technological changes a few years out and think through how to prepare for and exploit them. Obviously, that's not easy.

In his keynote address on NoSQL key value stores at the 2014 Velocity Conference, Brian Bulkowski discussed "in-memory" mechanisms and how they can be handy in electronic advertisements and similar situations. He mentioned during the keynote that "in-memory key value is Agile" is significant. This very thought opens up opportunities to consider the technologies of big data as enablers of business agility.

Emerging technologies and digital disruption will transform the enterprise, but they will also transform the ways in which we architect. What will this mean for enterprise architecture in general and for the role of the enterprise architect? How will EA help enterprises to collaborate with one another? What will these changes mean for the nature of the enterprise and its architecture? In this issue of Cutter IT Journal, our authors provide their practical insights and guidance on disruption and emergence and what they mean for EA.

To achieve pervasive, sophisticated, wide, deep, joined-up architecture descriptions, we need to develop a new generation of EA tools and techniques. The fact is that in an enterprise of any complexity, the architecture of roles, processes, applications, flow of work and data, data stores, and data sources quickly exceeds the ability of any of us to hold in our minds effectively. We need powerful ways to capture, map, navigate, and trace linkages, interfaces, and change initiatives.

I have mixed feelings about the FBI/Apple iPhone standoff. On the one hand, due to all the data collection and spying activities revealed by former US National Security Administration (NSA) contractor Edward J. Snowden, it seems like Apple has a pretty good argument for not wanting to comply immediately with a federal court order demanding that the company develop a specialized version of its iPhone operating system (iOS) that would enable the FBI to unlock the on-device security features of the iPhone belonging to one of the San Bernardino, California, terrorists.

Perhaps the best example of an adaptive and integrated field of knowledge is IT. Yet, it is quite shocking to see how many companies fail to invest a meaningful amount of time and resources to thinking actively about the future, with teams based in transdisciplinary collaboration.

In the days of large waterfall projects, organizations made the assumption that a software budget was allocated one-third per major category: analysis and design, develop, test. This was the rule of thumb that was used to generate a high-level estimate (HLE). This rule of thumb was great when it was applied evenly to all three categories. However, what usually happened on software projects is that the first two categories needed more time and it inevitably came at the expense of testing in an effort to stay on budget.

We are rapidly moving to a world where individuals don’t switch off their technologies, and companies can’t switch off their technologies. Head-up displays, image recognition, wearable technologies, virtual reality, a revolution in manufacturing technologies, the Internet of Things (IoT), super-dense computer memory … the list goes on and on! But what does this mean for the future of enterprise architecture (EA) — as a discipline, as a process, and as it informs the nature of an enterprise?

API-based development, with its focus on functionality rather than user interface, is a way for organizations to future-proof their investments in software. Requirements for functionality and interfaces keep changing more and more quickly, but, largely, what is not changing is the basic analysis and requirements that run the project. By outsourcing the interface to interested parties through your API, you can broaden the reach of your project (just as companies like Facebook, eBay, and Google have done).

Whether it is data warehouses or marts, data lakes, or reservoirs, the IT industry has a penchant for metaphor. The subliminal images conjured in the human mind by the above terms are, in my opinion, of critical importance in guiding thinking about the fundamental meanings and architectures of these constructs. Thus, a data warehouse is a large, cavernous, but well-organized location for gathering and storing data prior to its final use and a place where consumers are less than welcome for fear of being knocked down by a forklift truck. A data mart, on the other hand, creates an image of something between your friendly corner store and Walmart.

Business continuity planning, information security governance, and IT governance are critical activities in managing the operations of today’s organizations. Traditionally, these areas have been handled as separate and distinct activities. Effective organizations can no longer afford to manage these activities separately if they wish to streamline planning, guarantee adequate response to business-impacting events, and control costs.

It's time we step back from benchmarking specific topics and industries to consider the process of performance measurement and benchmarking itself. How should you use benchmarking information? What information should you gather about your company's own internal operations? In general, how should you gather and use performance data? It's important to look inward at your processes, your own ways of thinking, to make sure they still make sense and that you haven't developed any bad habits.