Introducing the API Economy: A Dialogue
by Jim Plamondon
This Executive Report introduces the API Economy to CxO-level executives of industries outside IT. It follows a novel dialogue format, as if it were a transcription of an introductory consulting session with the CEO and CIO of a fictional company in the oil and gas industry. The report defines API and the API Economy, discusses some of the many API business models, argues that this economy extends a proven software architecture onto the Internet, presents that architecture as a tool for thinking about business models, and discusses the role of developer relations in making APIs successful. The report then applies these ideas to the task of solving one of the fictional company's billion-dollar business problems using an API-based approach.
This Executive Report introduces the API Economy in a novel format: as a dialogue between the author (Jim Plamondon) and two C-level executives from OilCo, a fictional company in the oil and gas industry. We hope that this format will combine the richness and clarity of Cutter's traditional format with the "skim-ability" of a FAQ list and the human preference for stories, while also simulating the consulting experience more closely.
We chose to set the dialogue in the oil industry because that industry's current IT situation (including an explosion of data, an ongoing adoption of SOA and Web services, and an ever-increasing coupling of IT systems among partners, suppliers, and customers) is common in other industries, too. While the discussion does include a few technologies that are specific to the oil and gas industry (such as the WITSML data format standard), the patterns of problems and potential solutions are the same across most industries, even if the details differ.
The characters of our dialogue are as follows:
Jim: The author
CEO: Chief Executive Officer of OilCo, Inc.
CIO: Chief Information Officer of OilCo, Inc.
Jim: Thank you for inviting me here to OilCo to discuss the API Economy.
CEO: "API" -- you mean the American Petroleum Institute?
Jim: Good point -- I'm glad you pointed that out. I understand that you guys use about as many acronyms as the military. No, in this case "API" means "application programming interface."
Jim: I'll start by defining the API Economy and illustrating a number of API business models that have proven to be profitable. I'll then describe how the API Economy is extending a well-known software architecture to Web scale, and how that architecture can help you think about APIs and their business models. After that, we'll discuss the role of developer relations in building successful API ecosystems. Next, we'll discuss the role IT has played in the recent history of your industry. As an exercise, we'll develop a straw-man proposal for an API and its business model and explore the strengths and weaknesses of the proposal. Finally, we'll document a set of conclusions and next steps. We've only got an hour, so this will be a high-level strategic discussion, setting the stage for more detailed discussions later on.
Three points before we get started. First, I don't claim to be an expert in the oil and gas industry, its unique data, or its unique IT challenges -- although a colleague was kind enough to brief me on some of it. Instead, my expertise is in helping companies use the API Economy to achieve their business objectives, and particularly in helping companies accelerate the use of their APIs by external developers: what's generally called "developer relations" or "platform evangelism." When it comes to that, I'm an expert.
CIO: Fair enough.
Jim: Second, part of my role is to bring an outside perspective -- one that is not steeped in the culture and traditions of the oil and gas industry. This outsider's perspective has the advantage of being different. Not better, not worse, but different -- and a different perspective can help you see opportunities that you might otherwise miss.
Third, my role here is not to tell you what to do -- you two know the specifics of your industry better than I ever could -- and what you don't know, you have people who do know. My role is to help you explore how OilCo could use the API Economy to increase ROI and agility.
CEO: That's a big payoff, if it delivers.
Jim: Indeed it is; hence this meeting.
Information About Oil
Jim: Before I define the API Economy, here's a quote from Walter Wriston, the guy who built Citibank into a global powerhouse: "Information about money has become almost as important as money itself." We can paraphrase that as follows: information about oil has become almost as valuable as oil itself.
The oil and gas industry is now producing a gusher of information about oil. Joining the API Economy can unlock the value of that information.
CEO: But we're not in the information business. We're in the oil and gas business. Information technology is a means to that end, not an end in itself.
Jim: I totally agree -- and yet, won't you also agree that the importance of IT in the oil and gas business has increased dramatically in the last few decades?
Jim: Do you expect that trend to stop any time soon?
Jim: OK, then if one of your competitors had the opportunity to leapfrog ahead of you in their use of IT, would you be worried?
CEO: Mmmm ... maybe. Probably.
Jim: I would be, too. And that's what the API Economy offers: the opportunity to increase the value it derives from IT. Does that address your concern?
CEO: For now. Please continue.
THE API ECONOMY
Definition: The API Economy
Jim: The API Economy can be defined as follows:
The API Economy is the commercial exchange of information resources facilitated by:
Exposing your entity's core information resources to an ecosystem of developers through an Internet-based application programming interface (API).
Combining the information resources exposed by other entities' APIs to build new information resources.
Jim: One of the most important words in that definition is commercial. The commercial aspect of the API Economy is what makes it an "economy," not just a "system." The commercial aspect is encapsulated in a written agreement between the API provider and the API's customers. (An API's customers are the developers whose code calls the API). These agreements have various names. Some are called service-level agreements. Twitter calls its agreement the "Developer Rules of the Road." The point is that there's a deal -- a contract -- that defines the relationship between the provider of the API and the customers of the API.
CEO: Is the contract legally binding?
Jim: Yes, but the contracts almost always say that the API provider can change the terms of the deal in any way it wants, any time it wants, without notice -- so they are, in effect, unenforceable. Nonetheless, they define the mutual expectations between the API provider and the API customer, and that's important. One of the worst things that an API provider can do is change the terms of its contract arbitrarily, as Twitter recently did. This is just one potential cause of API failure, which I'll discuss more later on.
CIO: I noticed that you called the people who use our API "customers," not "consumers" or "users" or "developers." Why is that?
Jim: Because the word "customer" has unique connotations. People say that "the customer is always right," for example, but people don't say that the "consumer" or "user" is always right. I use the word "customer" to emphasize this point. Just as with any other product, an API needs to be designed, offered, and supported with the goal of delivering customer satisfaction as "job one."
CEO: I have a concern about this notion of "exposing" data. A lot of our data is proprietary, and controlling it is very, very important to us.
Jim: This is true for many firms in the API Economy. For every API that's exposed publicly, there are at least five that are exposed privately -- meaning within the originating company, or within that company and only a few select partners.
CIO: I have a different concern about "exposing" data. For instance, there are conflicting rules, internationally, about "data residency."
Jim: That's definitely a thorny topic, which I'm not going to be able to address today. Will it suffice, for this meeting, to say that this same issue affects your implementation of other IT initiatives such as the Digital Oilfield and Integrated Operations?
CIO: Yes, it affects them, too.
Jim: For now, then, let's just assume that the solutions to the data residency problems that you've found to make those other initiatives work will also work for the API Economy, and we can drill down into counterarguments and workarounds later. Fair enough?
CIO: For now, OK.
CEO: I've got a question. If it's an "economy," how does one make money from it?
Jim: It depends, and that's a great segue to the next topic, "API business models."
API Business Models
Jim: If something is "commercial," then it needs a business model. To show this, I've borrowed a number of charts from John Musser of ProgrammableWeb (with his permission, of course). Figure 1 shows the API business models that existed seven years ago, in 2005, when the API Economy was just getting started.
Jim: Figure 2 shows the API business models that were known to exist as of late 2012. As you can see, there has been an explosion of API business models since 2005. The 2012 chart shows so many different business models that its text may be too small to be readable. This explosion isn't over yet. In fact, I'm going to add to it before this meeting is over.
We don't need to discuss all of these business models here today. Instead, we'll discuss just enough to get a sense of their variety and to understand why they differ.
When thinking about API monetization, remember that it's a two-sided issue. You need to monetize your side of the API somehow, but so do your API's customers. Everyone must win, or no one will win.
Free (Best Buy)
CEO: I see "free" on the chart, over at the left. How can "free" be a commercial business model?
Jim: The quick and easy answer is "advertising." That's how Facebook does it, anyway.
However, there are other ways to monetize a free API. One of the simplest is when your API opens up new distribution channels for stuff that you're already selling.
One example is Best Buy. It's a top US retailer of electronics. Like all other retailers, it's had an e-commerce website for over a decade, enabling customers to order products online. A few years ago, however, it started adding APIs to enable other websites and mobile apps to see what Best Buy was offering, at what price, and in what stores. This created a whole new indirect distribution channel for its products. It also enabled new business opportunities, such as partnering with other companies' loyalty programs. If you have loyalty points with Citibank, for example, you can redeem them instantly for products at Best Buy. Everybody wins: Citibank customers have less hassle and more options when redeeming loyalty points, Citibank automates its point-redemption system, and Best Buy sells more product, which is how it monetizes its free API.
CEO: OK, so free APIs work for advertising and for new distribution channels. Is that all?
Jim: Free APIs are often offered to complement a paid service. Let's talk about them next.
Developer Pays: Pay As You Go (Amazon Web Services)
Jim: Here's the "developer pays" part of the business model hierarchy. As you can see in Figure 3, there are a lot of variations on the "developer pays" model. I'm only going to discuss three of them here today: pay as you go, transaction fee, and freemium.
In cloud computing, such as Amazon Web Services, you can create and operate a virtual data center by calling APIs. You call one API to spin up a server, another API to store a file, another to activate a load balancer, and so on. The cloud service provider may have a browser-based control panel, too, but it calls the same APIs. Therefore, at their foundation, cloud computing services are API-based.
Every cloud service has a slightly different billing pattern, but generally speaking, you pay for the service activated by the API calls, not for making the API calls themselves. For example, if you call the API that spins up a virtual server, then you pay for every hour that the server is operational (and the size of the server). If you call the API that stores data, then you pay for every hour that you're data is stored (and the amount of data). And so on. This is the "pay as you go" model.
CIO: Yes, our staff keeps using Amazon for little side projects. Lately, its been using it for some bigger projects, too, like rendering animations of reservoirs. So I'm familiar with that.
CEO: I see PayPal on the chart. I use that. What is its API business model?
Developer Pays: Transaction Fee (PayPal)
Jim: Developers can call PayPal's API for free, but every time someone gets paid using PayPal, a little piece of the action goes to PayPal in a transaction fee. Fundamentally, it's the same situation as with Best Buy: the API is just a new distribution channel for its existing products. It doesn't need to charge for the API, because it can charge for the things sold through the API. Best Buy sells tangible goods, whereas PayPal sells an intangible payment service, but fundamentally, the model is quite similar.
CEO: Is the "developer pays" model common?
Jim: Yes, very common -- although it's often combined with a free service, producing the "freemium" model.
CEO: What's that?
Developer Pays: Freemium (Google Earth)
Jim: The freemium model is the most popular of API business models. It combines a stripped-down "free" service with a full-featured "premium" service: hence "free-mium." The free API helps to extend the API's brand and to lock the market's developers into your API, while the premium API monetizes the value thus created.
CEO: What's an example?
Jim: Well, you guys are familiar with Google Earth, right?
Jim: So you know that Google Earth has a free version and a paid version: Google Earth Enterprise. That's a classic freemium model. The premium service can be paid per use, per volume, per month, or in any number of different ways, as appropriate for the API provider and API customer.
Developer Gets Paid: Rev-Share (Google AdSense)
Jim: Now let's look at models in which the API's customers -- developers -- get paid for using an API (see Figure 4).
There are a lot of different models here, but time is short, so I'm only going to discuss one: Google's AdSense program, which uses a kind of revenue sharing.
AdSense and AdWords are the opposite sides of the same coin. In AdWords, advertisers bid for keywords they care about, like "gasoline" and "diesel." In AdSense, publishers' websites display ads that are related to their content, as determined by Google. If your website has a lot of information about diesel, then Google will funnel ads to it that are related to "diesel." Then Google will pay you some tiny portion of the money it collected from AdWords for the word "diesel."
The bottom line is that, by calling the AdSense APIs for free, you can get paid for the ads delivered to your site.
CIO: But your website is cluttered with advertising.
CEO: Same as newspapers and broadcast TV, but that's what makes the content free for consumers.
CIO: Fair point.
Indirect: Mobile/Devices (Netflix)
Jim: The final top-line category of API business models is "indirect" (see Figure 5).
As before, I'm going to focus on just one of the above monetization models: in this case, Netflix's "mobile/devices" business model.
CIO: I love Netflix. Sometimes, when I'm traveling, I'll call my wife and we'll watch a movie together even though we're a thousand miles apart.
CEO: I didn't know you did that. That's a good idea. I should try that.
Jim: I see you [CEO] have an iPhone, and you [CIO] have an Android, right?
Jim: And your wife watches Netflix on?
CIO: The Xbox.
Jim: Well, that's what Netflix's API business model is all about: streaming video to any client device. Last I heard -- and this number goes up constantly -- Netflix was serving its content to more than 800 different devices. All of these devices access Netflix's cloud-hosted video files, viewer ratings, suggestions, etc., through Netflix's APIs. Although Netflix is an extreme case, it is very common these days to serve up cloud-hosted content on a wide variety of devices. The mobile-cloud nexus is extremely powerful.
CIO: Did Netflix write all of those clients itself, or did external developers write them?
Jim: When Netflix launched its API in 2008, it expected that its API would be used primarily by external developers. But that's not what happened. Instead, Netflix itself wrote more or less unique presentation layers for every client, but relied on its API to access its cloud-hosted data layer services uniformly. More than 90% of the traffic through its API comes from clients that Netflix wrote itself.
CEO: So how does it monetize its API?
Jim: Indirectly, by reducing the cost of developing new clients and increasing the agility with which new business opportunities can be embraced. There's no way it could have developed, updated, and maintained 800 clients by now if it hadn't used an API to encapsulate its back-end services.
CIO: I get it. I also notice that there's an indirect model called "SOA+," with Amazon as its example. Amazon was also an example for one of the "developer pays" models (pay as you go) and for one of the "developer gets paid" models (affiliate). What's up with that?
Jim: You've got to give Amazon credit. It has been an innovator all across the board. Part of the reason is that it has a super-flexible internal computing infrastructure due to an early and comprehensive commitment to service-oriented architecture (SOA). All of its data and processes are already exposed, internally within Amazon, as services: that's "SOA+." Technically, all it needs to do, to expose one of these services publicly, is flip a switch. That's an oversimplification, of course, but you get the point. Likewise, Amazon had already run a lot of different partner and affiliate programs before it started exposing APIs, so it recognized that one business model didn't necessarily fit all services. Amazon crafts each API and its business model together, more or less, to serve the needs of its business and of its customers' businesses.
CEO: Not bad for a bookseller.
Jim: Not bad at all. But that's the thing, of course: Amazon never saw itself as being "a bookseller." It saw itself as a seller, with books simply being its first product. Likewise, the oil and gas industry should not see itself simply as being an "oil producer" or even as an "energy producer." It's an information producer, too. It has a lot of information to monetize, through one business model or another.
Business Models Summary
CEO: Let me summarize this back to you, to make sure that I've got your point. Bottom line is: an API and its business model need to be designed together to serve both the API provider and the API customer, but there are a lot of different API business models to choose from and lots of room for new business models, too.
Jim: That summary works for me.
CEO: We're about a third of the way through our allotted time, here. Next topic?
Designing API Business Models
Jim: I've just shown you a bunch of business models organized into a taxonomy -- that is, a hierarchy of shared characteristics. That's useful, because it helps show the wide range of business models that have already been explored. But it doesn't help you design a business model that's right for you.
Steve Willmott at 3scale developed a way of thinking about API business models that I like. It leverages the model-view-controller (MVC) architectural pattern (see Figure 6).
CIO: Yes, that looks familiar. I see that you've also mapped the three-tier architecture to MVC: data, form (I call it "presentation"), and business logic.
Jim: Same general idea. The API Economy is extending the MVC architecture to the Internet. Some APIs manage data, some transform or control that data through business logic, and some integrate and visualize that data.
The different parts of the MVC model enable different business opportunities, as shown in Figure 7.
Jim: Data API examples include the AccuWeather weather API, the New York Times news API, and the Bloomberg financial data API, among thousands of others.
One great "view" API is the Netflix API, which we already discussed. Netflix's API enables Netflix's video data to be rendered on hundreds of different client devices. Each client is, in effect, a different "view" of the video data.
CIO: Wait, isn't Netflix a "data" API? It's just exposing video data, after all.
Jim: Good question! The thing is, Netflix doesn't own the video data. That's owned by the movie studios. Netflix just offers a new way to view that data -- online, on any device. So it's a view API, not a data API.
CIO: That seems like a distinction without a difference.
Jim: Ultimately, every API is a data API because it's accessing and perhaps altering the state of some data somewhere. The difference is in the API's intent. The intent of the Netflix API is to enable the viewing of video content on every device, everywhere. That's a viewing "intent," even if it (inevitably) requires sending and receiving some data.
CIO: OK, I can live with that.
Jim: Likewise, there are a lot of "controller" APIs. Consider AdSense, which we discussed earlier. It controls which ads get placed on which webpages. Or the various cloud computing APIs, which control virtual servers, load balancers, and network switches, for example. Yes, they send and receive a little data, but their intent is to control.
CEO: How does this MVC approach help define an API business model?
Jim: By providing a framework for thinking about any new API you might offer. Later on, I'm going to propose -- just as a thought exercise -- that you consider exposing a particular real-time data stream through an API.
CIO: Which data stream?
Jim: I promise to tell you before the meeting's over, but for right now, it doesn't matter. The point is that it's clearly a "data API." What the MVC architecture tells us is that we'd want to make it easy for other people to create new views of the data that they get from our API. Likewise, we'd want to make it easy for other people to transform, with additional business logic, the data that they get from our API; that is, to make it easy to write new controllers. Understanding that our API is a data API makes it easier to think through the business model, the potential partnerships, and so on -- all of which will loop back and influence the design of the API itself.
CEO: Let's assume that we develop a great API, put a great business model around it, and share it on the Internet. Are we done?
Jim: No. For new APIs, the leading cause of failure is the "build it and they will come" idea. To get customers to start using your new API, you'll need to develop a number of additional resources and engage in a number of additional activities, which are collectively called "developer relations." This is true whether your new API is internal-only, shared only with a few close partners, or shared with everyone on the Internet. The scale of what you need to do changes with the scale of the API's use, but the basic resources and activities are always the same.
CEO: Tell me more about developer relations.
Jim: You'll recall the definition of the API Economy, from earlier in this meeting. It includes the idea that the API is exposed to "an ecosystem of developers." That's true whether those developers are all internal to your company, or also include close partners, or also include every developer on the Internet.
Fundamentally, the role of developer relations is to make participating in your API's ecosystem so awesome that developers choose to join it, and to stay in it, even when other awesome APIs and ecosystems emerge.
CEO: What's an "API ecosystem"?
Jim: In brief, an API ecosystem is the collection of interdependent people, things, and processes that use an API.
CEO: Can you give me an example?
Jim: Sure. Consider Apple's iOS ecosystem. Apple makes the iDevices (including the iPod, iPhone, and iPad), the iOS operating system, the iOS app development tools, and some iOS apps. Telcos carry the iPhone's signal. Independent developers write iDevice apps. The common thread is the API that's exposed by Apple's iOS operating system. It's what ties Apple's iOS ecosystem together.
Apple's iOS ecosystem competes primarily with Google's Android ecosystem, centered on the API of the Android operating system. It also competes with RIM's declining BlackBerry ecosystem and Microsoft's struggling Windows Phone ecosystem.
CEO: We use Schlumberger's Ocean and Petrel software. Is it an ecosystem?
Jim: Absolutely. It has a well-defined API, good documentation, a store for buying plug-ins -- it's a very solid ecosystem. It exposes a number of APIs, such as the ones accessed by its mobile InterACT apps. Schlumberger is further into the API Economy than just about anyone else in the industry, aside from some much smaller start-ups. I'd like to help you catch up.
CEO: But Schlumberger is a software company, among other things. We're not.
CIO: True, but with the MVC architecture being extended to the Web, we don't have to be a software company to benefit from the API Economy. We can just expose some data APIs, for example, and let other companies build the controller and view software.
CEO: I'm not entirely convinced, but let's move on.
Resources, Activities, and Beliefs
CEO: How do you make an API ecosystem "awesome"?
Jim: In brief, with a combination of resources, activities, and beliefs -- backed up by a business model that's compelling for API customers and sustainable for the API provider -- that leads all members of that ecosystem to feel that they are valued members of a winning team on an inspiring mission. (I borrowed that latter phrase from Graham Weston, chairman of Rackspace.)
"Resources" include things that help developers get started and work efficiently, such as documentation, sample code, discussion forums, and adapters for their favorite tools.
"Activities" include anything that injects positive energy into the API ecosystem, such as responding to questions, sponsoring pizza at Meetups, and recognizing community MVPs.
"Beliefs" include the inspiring mission you're engaged in and the broad strokes of how you expect the ecosystem to achieve that mission.
CEO: Beliefs? Seriously?
Jim: Absolutely. Consider Apple. According to Steve Jobs, Apple's core belief is that "people with passion can change the world for the better." 1 It's hard to imagine a more inspiring mission because it embraces all inspiring missions. By association, Apple's customers believe that buying Apple products signals that they are "people with passion to make the world a better place." This belief is what let Apple survive its near-death experience in the mid-1990s, and this belief is what has recently led Apple to become the most valuable company in the world.
CEO: It didn't hurt its comeback to have three great products in a row: the iPod, iPhone, and iPad.
Jim: It's easy to confuse the effect with the cause. I would argue that Apple produced those three great products because of its core beliefs, not the other way around.
CEO: Let's keep this on track, guys. Anything else we should know about developer relations?
Jim: Yes -- that doing developer relations incorrectly is the leading cause of failure among public Internet APIs. I don't have any data on whether or not it's also the leading cause of failure among private APIs, but it seems likely. So whether your API is internal-only, partner-only, or public, you still need to implement a developer relations program.
Needed for Internal-Only APIs?
CIO: Why would we need to invest in developer relations for an internal-only API?
Jim: Good question. Developer relations provide the basic resources that every developer needs: API documentation, sample code, discussion forums, and the like. If the API is internal-only, then your internal developers can't turn to outside resources for help because there aren't any outside resources. They can't expect to get their questions answered on public discussion forums such as StackOverflow, for example. They are utterly dependent on the resources that you make available internally.
Likewise, in any enterprise-sized company, internal communications across groups tends to be imperfect. Your developers are more likely to know what's hot in Facebook's API ecosystem than in their own company's internal API ecosystem. So you need to invest in the resources, activities, and beliefs that make it easy for your internal staff to become aware of your APIs, to adopt them, and to improve them over time. That's developer relations.
Developer Relations Summary
CEO: In summary, if we're going to invest in exposing an API -- even an internal-only API -- then we should also plan on investing in developer relations to build and maintain the API's developer ecosystem. Correct?
CEO: OK, so, moving right along -- how do we apply all of this in our company?
Jim: To answer that, let's start by taking a quick walk down memory lane to see how we got to where we are today. I'm going to focus on the exploration and production (E&P) side of the oil and gas industry.
THE JOURNEY TOWARD THE API ECONOMY
Different Industries, Same Pattern
Jim: Although this brief history is focused on oil and gas, the same pattern of sociotechnical progress toward the API Economy is found in other industries. Each industry has taken lessons from the industries that executed the pattern earlier. For example, lessons from the manufacturing industry's computer integrated manufacturing (CIM) vision were used in the development of the oil and gas industry's Digital Oilfield vision. 2
CEO: The Digital Oilfield vision has led some companies to invest a lot of money wiring up oil wells with sensors, but it's not yet clear that it brings clear-cut financial benefits.
Jim: You'd know better than I -- although I've seen a number of papers and articles in just the last few months suggesting that the industry's leaders are starting to see some serious economic returns on that investment. Can we come back to that?
Jim: Starting in the 1980s, the amount of digital information being gathered about subsurface geology -- "what the Earth looks like below the surface, where oil and gas lives" -- started to grow rapidly, with data from well-bore logging, seismic surveys, and the like.
Software was developed to make sense of all this data, eventually merging into software suites such as Schlumberger's Petrel, Halliburton's Petris, and Paradigm. Initially, all of these software suites were proprietary (i.e., not open source) and most still are -- although I understand that Schlumberger's Ocean platform is now open source. Also, they are based on Microsoft's proprietary Windows platform (i.e., not the open source LAMP stack). So far, so good?
CEO: What's the "LAMP stack"?
CIO: The stack of open source software technologies that powers most of the Internet: the Linux operating system, Apache Web server, MySQL database, and some programming languages that start with 'P'. Linux, Apache, MySQL, P-language -- L-A-M-P, LAMP.
CEO: Microsoft Windows is not in the LAMP stack?
CIO: No, the LAMP stack is the open source alternative to Windows.
CEO: We're using Windows, right? Does the API Economy require this other stack?
Jim: The API Economy works just fine with either Windows or the LAMP stack. No problem.
Lost in Translation
Jim: Each such software suite -- and often, individual applications -- used unique and often proprietary data formats to store and communicate data. That made it hard to get data to flow smoothly from one application to another, across different application suites. The result was vendor lock-in. When two or more companies, using incompatible suites, tried to work together on a joint venture, it was a nightmare. Translating the data back and forth between formats tended to be expensive and had the risk of having valuable insights be "lost in translation."
Another issue is scale. The size of companies in the E&P industry runs the gamut from tiny explorers to the largest Fortune 500 companies. The software that's ideal for Exxon Mobile tends to be too expensive and complicated for a tiny independent company. Different software companies tend to serve customers at different scales, often using different data formats.
Data Interchange Formats
Jim: In the 1990s, a number of oil companies got together to form the consortium that's now known as Energistics, to define and promote the use of cross-industry data standards. The results include WITSML for well data, PRODML for production data, and RESQML for reservoir data. These XML schemas have been widely adopted for data interchange.
CIO: Widely, but not universally. RESQML is still pretty new. Most software supports only the older RESCUE format, also from Energistics.
Jim: Which is not surprising, because this stuff is hard. Just look at the history of the screw threads on nuts and bolts, for example. They had to be standardized to be interchangeable. You might think that the amount of information needed to completely specify a screw thread was small, but you'd be wrong. Even though screw-thread standardization began around 1800, it took almost a century for international standards to emerge. As recently as the 1990s -- two hundred years after screw-thread fasteners were first mass-produced -- noncompliance with international standards became a problem for US manufacturers who imported fasteners from overseas, leading to a boom in the domestic US fastener industry. 3
CEO: What does that have to do with oil and gas?
Jim: Your industry has had to cram two hundred years of standardization into just a few decades. It's not surprising that it's been hard. What's surprising is that it has gone so well. A lot of IT staff in the oil and gas industry deserve a pat on the back for a job well done.
CEO: That's an interesting perspective. Our IT guys are always asking for more money, and it often seems like they want to use the latest technology just for the technology's sake ... no offense.
CIO: None taken.
CEO: It's encouraging to hear someone from the outside say that it hasn't been in vain.
Jim: Looking in from the outside, it was definitely worth it -- and it gets even better.
Jim: In the early 2000s, the Digital Oilfield concept caught the industry's imagination. Now, I know that you know what the Digital Oilfield is, but just to make sure that we're on the same page ...
In a Digital Oilfield, the state of every thing and every process in an oilfield -- drill strings, wells, pumps, pipelines, valves, reservoirs, everything -- is continuously monitored and analyzed in real time, with the data used to continuously improve a digital simulation of the oilfield. Insights gained from the simulation are used to control things and processes in the real oilfield. For example, if data from a drill head indicates that it has hit a given subsurface rock formation at a shallower depth than the simulation predicted, then this new data can be used to update the simulation of the subsurface geology, and this report may prompt a change in the drill's direction. Likewise, analysis of the simulation may suggest that an expensive piece of equipment may be on the verge of breaking down, causing a repair order to be issued in the real world.
We're seeing this same pattern in just about every industry. For example, the devops movement in software development has very similar aims and accomplishes them through remarkably similar sociotechnical means.
CIO: You keep using that phrase, "sociotechnical." What does that mean?
Jim: It's a buzzword. It means that "neither technology alone, nor social systems alone, are enough to effect change." You almost always need to change both, and the changes loop back into each other. 4
CIO: We've found that the social systems are usually the hardest to change.
Jim: Yes, that's often the case. That's why I spent most of my career exploring how to accelerate the adoption of new technologies. Even when a technology works perfectly, nonadoption can make it fail.
CEO: Let's stay focused here. You mentioned the Digital Oilfield and Integrated Operations. Does that bring us up to the present in your history?
SOA and Web Services
Jim: In the early 2000s, around the same time that the Digital Oilfield vision was catching on, the industry also started moving toward an SOA using Web services. It's been a long haul, but with the Energistics' data interchange formats, the software modularity enabled by SOA and Web services, and the social/managerial adaptations of Integrated Operations, the Digital Oilfield is finally becoming a reality.
CEO: Maybe so ... but is it ever going to add value to the bottom line?
Jim: It's my understanding that it has, at least for the market leaders. I have seen a spate of articles, in just the last few months that attribute massive cost savings to the emergence of the Digital Oilfield. 5, 6
One particular example stands out: Chevron -- which the press suggests is leading the industry in putting the Digital Oilfield concept into practice -- is also leading the industry in earnings-per-share growth, margins, return on capital, total stockholder returns, and just about every other financial measure. 7 Chevron has stated that its implementation of the Digital Oilfield gives it a "sustainable competitive advantage." 8
CEO: Is Chevron the only one profiting from its investment in the Digital Oilfield?
Jim: I don't think so, although again, I'm not an expert in the field. A colleague suggested that Saudi Aramco has invested more in well instrumentation than just about anyone ... and it is doing particularly well financially, too. I can't find any proof that Saudi Aramco's recent financial success stems from its Digital Oilfield investments, but it's worth noting that Saudi Aramco is the cochair of the SPE Intelligent Energy International conference in Dubai next year (2013).
CIO: So you're saying ... we're done? The industry leaders, at least, have reached information nirvana?
Jim: Hardly! What I'm saying is that the emerging realization of the Digital Oilfield vision enables the oil and gas industry to take the next step.
CIO: Let me guess: this new step will involve the API Economy?
Summary of the Brief History
CEO: Before we move on, let me make sure that I've grasped the key points of your little trip down memory lane. You're saying that the information explosion that started in the 1980s led to a slew of incompatible data formats that have now been standardized, more or less, and that the use of these standards in SOA-based Web services positions us well to move into the API Economy?
THE API ECONOMY IN OIL AND GAS
CEO: Now that we know what the API Economy is, how should we get involved, in your opinion?
Jim: Ultimately, I can't answer that question, but I can help your staff answer it. Here are the pertinent steps to consider: (1) identify a business opportunity that could be efficiently addressed using APIs; (2) design an appropriate business model; (3) design a developer relations plan and train your staff to execute it; and finally (4) design and deploy the API.
CEO: I can see that we're not going to be able to cover all of that in what's left of our time today.
Jim: Indeed. However, you'll recall that I promised to describe a new business model before this meeting was over -- one that wasn't already on the big API business model chart.
CEO: OK, we have time for that, if you make it quick.
Jim: Will do.
Caveats on this Example
Jim: First, a caveat. The example I'm about to give is somewhat a shot in the dark. But hopefully it will make the point that an API and its business models should be designed to solve a business problem, for both the API provider and the API customer. Whether the following combination of API and business model would actually work is irrelevant; it's just an example of the process. Fair enough?
CEO: Fair enough.
Jim: With that caveat in mind, let's start by describing the business problem that we need to solve.
Jim: The problem this example will address is the high cost of equipment failures and preventive maintenance.
CEO: That's a problem, all right.
Jim: Anyone who watched the US news in 2010 will remember the huge oil spill into the Gulf of Mexico, which was caused by a cascade of equipment failures, eventually leading to the failure of the blow-out preventer (BOP).
CEO: It wasn't just the BOP that failed. The reporting and decision-making processes also failed, and there were other errors in well design and elsewhere that contributed.
If those other failures hadn't happened, then the BOP wouldn't have failed, and the spill would never have happened. A dozen lives and tens of billions of dollars' worth of environmental, social, and economic damage would have been avoided.
Also, because of the Gulf oil spill, the cost of insuring offshore oil rigs has gone up by $10,000-$15,000 per day. With nearly 700 rigs under contract around the world right now, that's another $3 billion a year of insurance costs alone.
CIO: The Texas City gas plant explosion in 2005 was caused by a similar combination of equipment and process failures. It cost many lives and over $1 billion dollars in damages.
Jim: Can we agree that this is a billions-per-year problem?
Jim: On a smaller scale, I understand that monitoring a single huge natural gas compressor for impending mechanical failures can save millions, too.
CEO: That's right. If a gas compressor breaks unexpectedly, then all of the gas wells feeding into the compressor go out of production until it's fixed. The lost production can cost millions. Even worse, it can disrupt the entire value chain, from the wells right through to final customer delivery.
CIO: As you can imagine, we invest a fortune in preventive maintenance, so that unexpected failures are rare.
CEO: OK, Jim, you've established that equipment failure and maintenance is a huge cost. What's your proposed solution?
Proposed Solution (API)
Jim: My proposed solution has two parts: an API and a business model.
The API would expose the real-time data stream regarding the current state of your equipment and of any decisions regarding that equipment, such as its maintenance schedule. I suggest that the API be public from the start. Initially, only a small subset of your equipment would be exposed this way, but the goal would be to expose all of it.
CEO: What good would that do? Exposing the equipment's current state won't make it run any better.
Jim: No, but it can improve the software that predicts impending equipment and process failures.
Jim: Do you, like Chevron, already gather real-time data about your equipment's condition and use data mining and artificial intelligence to schedule "predictive maintenance"? 9
CIO: Yes. We have an internal Web service for real-time equipment-condition data, and we've got some experts writing some pretty fancy software for making failure predictions from that data.
Jim: OK, then here's the key point: opening up the data stream can produce better prediction software, and better predictions can save you a fortune -- possibly billions per year.
CEO: I don't know about that. These gas compressors are highly specialized devices. It's not as if some kid in a garage can create prediction software that our experts can't.
CIO: Actually, Jim's got a point. It's crowdsourcing. Get a big enough crowd together, and they can find better solutions than any small group of experts.
CEO: What, like a million monkeys eventually typing out Shakespeare?
CIO: No, more like using a crowd of independent software developers to write Linux, the open source operating system. My Android phone is based on Linux; so is Amazon Web Services. Is that what you mean, Jim?
Jim: Yes, exactly. By opening up the compressors' data stream to the public, we can crowdsource better software for predictive maintenance. Besides, I'm not talking about kids in a garage, so much as I'm talking about insurance companies, academic researchers, and independent specialists whose experience crosses lots of different industries -- not just oil and gas.
CEO: Why would they bother? What's in it for them?
Jim: That brings me to the second part of my proposed solution: the API's business model.
Proposed Solution (Business Model)
Jim: Let's go back to our MVC-based approach to analyzing potential business models. As shown in Figure 8, our real-time equipment-state data stream API is a data API, indicated by the box at the top labeled "Model (Equipment Condition)." Our billion-dollar need is for "better controllers" -- that is, software that can more accurately predict impending failures based on the data that they get from our API. That's indicated by the box at lower right, labeled "Controller (Failure Prediction Software)."
CIO: What about "views"?
Jim: I can imagine many different "view" apps for different client devices, from text message alerts to the kind of real-time visualizations used in the financial markets, but I'm going to propose a very different means of "viewing" failure predictions.
Wisdom of Crowds
Jim: According to James Surowiecki, who wrote The Wisdom of Crowds, four conditions must be met for a crowd to be smarter than any individual expert: 10
1. Diverse. The crowd must be diverse, so that people are bringing different pieces of information to the table.
2. Decentralized. The crowd must be decentralized, so that no one at the top is dictating the crowd's answer.
3. Independent. The people in the crowd must be independent, so that they pay attention mostly to their own information, not worrying about what everyone around them thinks.
4. One verdict. The crowd needs a way of summarizing people's opinions into one collective verdict.
Jim: Does your internal failure prediction team meet all of these criteria?
CIO: No. We've only got a handful of people on this, and that's not enough to be "diverse." They're all working together, under the same management, so they're not decentralized. The collective verdict is issued by their management, which meets the "one verdict" condition but violates the "independence" condition. We also have a contract with AzimaDLI. It has another fortyish experts who help monitor our equipment-condition data stream and assist with predictive maintenance.
Jim: That's still a pretty small "crowd," and it is spread across only two companies, which violates the "independence" and "decentralized" conditions.
CEO: So what business model can meet all these conditions?
Jim: A prediction market.
CEO: I've heard of prediction markets. They accept bets on things like elections, right?
Jim: Yes. More precisely, they're an extension of derivatives trading -- such as trading in crude oil futures -- to everything else. They've been proven to be better than individual experts at predicting all sorts of things. 11
CEO: How would a prediction market work for gas compressor failures?
Jim: "Predictive maintenance" assumes that you can predict when and how a given piece of equipment will fail. In a prediction market, people compete to predict such failures with the most accuracy. The more accurate their predictions, the more they can bet on each prediction, and the narrower the prediction window.
CIO: Can you give me an example?
Jim: Sure. Let's say that a particular compressor's next regularly scheduled preventive maintenance is on 1 June 2013. That's when the manufacturer suggests that it be serviced, all else being equal. With an open data stream and a prediction market, predictive maintenance specialists like AzimaDLI could put its money where its mouth was.
For example, if AzimaDLI's analysis of your equipment-prediction data stream suggested that a given compressor was on a path to fail on 15 March, then Azima could place its bets accordingly. Other experts in predictive maintenance, using different prediction algorithms, could come to different conclusions and predict failure on different dates.
CEO: And what -- we'd pay off the winning bets?
Jim: No. If you bet on the winning Super Bowl team, its quarterback doesn't owe you money. The winner collects, indirectly, from the people who bet wrong. Same in the commodity futures markets and same in prediction markets.
Now, you could jump-start the market by placing bets on the scheduled maintenance date -- or, even better, getting the compressor's manufacturer to do so. When the compressor is actually serviced -- or fails -- those who bet that date win, and everyone else loses. The bets could be more complicated than that, but that's the basic idea.
CEO: Do we get a piece of the action?
Jim: You get the most valuable piece of all: the aggregated "wisdom of crowds" as to when the compressor is most likely to fail. You can use that wisdom to deliver "just in time" maintenance to the compressor.
CEO: Wait a minute. What about insurance? We pay a fortune to insure our operations against unexpected failures.
Jim: Buying insurance is very similar to placing a bet in a prediction market. I suspect that insurers would be very interested in tracking failure predictions and adjusting their premiums in real time. Insurance companies could be major bettors in the failure prediction market.
CIO: Does a prediction market meet all four criteria you mentioned?
Jim: Yes, so such a market should be better than any individual expert, or small group of experts, at predicting equipment failures, all else being equal.
CIO: Let's imagine that this prediction market was in place. How does this business model meet our API customers' needs?
Jim: What they need is a means of monetizing their ability to make accurate predictions of equipment failure. They could do this in two ways.
First, if the amount of money involved in the prediction market was high enough, then the people with good predictive software could make a living by winning the bets placed by those with inferior predictive software. That's how the financial markets work these days, via "program trading."
Secondly, the success of a given piece of software in the prediction market could be used to encourage other firms to license the software's prediction stream, which would be exposed through a paid API.
CIO: That could work.
CEO: If prediction markets are so spiffy, why isn't everyone using them?
Jim: For two reasons. First, they require a real-time data stream. Until recently, these have only existed in a few markets, such as finance and weather. However, the API Economy is starting to expose a lot more data streams, so prediction markets can now start emerging elsewhere.
Second, prediction markets are illegal in the US, because they are considered to be "online gambling," which was banned by the 2006 Unlawful Internet Gambling Enforcement Act.
CEO: That is, indeed, a significant impediment.
Jim: Nonetheless, there is a legal US-based prediction market that could be used to test the proposed business model: Iowa Electronic Market. It got a "no action" letter from the relevant federal regulatory body, allowing it to legally operate.
Also, there's a strong movement afoot to repeal the ban. If that movement had the backing of the oil and gas industry, it would be much more likely to pass.
CEO: Given this legal impediment, are you seriously suggesting that we set up a prediction market for equipment failures?
Jim: No. That's why I put a big caveat at the start of this discussion. The purpose of this example was to identify a known business problem and walk through the process of identifying a potential business model for its solution. I also wanted to demonstrate that the range of potential business models was very wide. Would this proposed business model work? Only if prediction markets were legal and widely used, which they are not, currently. Still, given the magnitude of the problem -- billions per year -- it isn't completely outside the realm of reason.
Potential Business Model Summary
Jim: This example showed the process of identifying a serious business problem (the high cost of equipment failure and maintenance); using the MVC model to identify the core asset (the equipment-state data stream) and its complements (failure prediction "controller" software and failure prediction "view" software); proposing a business model; looking at that business model from the API provider's and API customer's perspectives; and exploring any legal, regulatory, or other impediments to the potential success of that model.
CEO: You also stated that entering the API Economy could increase agility. How so?
Jim: There are lots of great examples of this. One is Best Buy. After Best Buy exposed its catalog through an API, Visa integrated Best Buy's catalog and order-placement system into Visa's loyalty rewards program. Now, members of Visa's loyalty rewards program can instantly and conveniently redeem their points for Best Buy products. Citibank and Chase followed suit. Best Buy didn't create its API with rewards programs in mind, but having the API made it easier for Best Buy to seize the opportunity.
Another example is Twilio. Twilio's API makes it easy for other applications to send text messages and place Internet-based phone calls. A developer at Intuit wanted to add a security feature to Intuit Online Payroll: if someone tries to change your account info, then Intuit sends your phone a text message asking if you are the person making the change. If not, the changes are discarded. Using Twilio's public API, the Intuit developer was able to add this feature over a weekend, using a personal credit card to set up Intuit's Twilio account (later shifted to an appropriate corporate card). Intuit got a great new feature, and Twilio got a great new customer, with no executive negotiations or other traditional "business development" overhead. That's agility!
CEO: How would our agility be increased by exposing the equipment-state data we discussed above?
Jim: Let's say that someone in the power-generation industry made a breakthrough in failure prediction software, based on vibration analysis, lubricating oil analysis, or some such -- the same kinds of data exposed through the API that I proposed. If your data stream was publicly exposed, then that breakthrough could be applied to your data without any involvement by you. You wouldn't even need to know what they breakthrough was; it could be entirely proprietary to the person who made the breakthrough. All that you'd see was a sudden improvement in the accuracy of the failure predictions in the proposed prediction market. You'd benefit without lifting a finger. It's hard to get more agile than that.
CIO: That's just the benefit of modular interfaces.
Jim: Exactly right. The API Economy extends the benefits of the modular MVC architecture to Web scale.
CEO: Looks like we're out of time. Where should we go from here?
CIO: I'd certainly like to take the next step. I'm not convinced that it will produce an earth-shattering revolution, but I don't want to miss the boat, either. There's never enough time for things like this, but we can squeeze it in somehow.
CEO: All right, let's do it. Keep me informed.
Jim: Thank you for your time here today. I wish you well!
In closing, we hope you've enjoyed this dialogue format as a break from the typical style of an Executive Report.
1 "Steve Jobs About Apple's Core Value" (Video). YouTube, 1997 (www.youtube.com/watch?v=dR-ZT8mhfJ4).
2 Nguyen, Paul. "The Road to an Upstream Reference Architecture." Microsoft, 2010.
3 "The History of the U.S. Fastener Industry." Ocean State Stainless, Inc. (www.osstainless.com/articles/fastener-history.php).
4 Rao, Hayagreeva. Market Rebels: How Activists Make or Break Radical Innovations. Princeton University Press, 2008.
5 Hatch, David. "At Chevron, Digital Technology Spouts Gusher of Savings." US News and World Report, 12 June 2012.
6 Leeber, Jessica. "Big Oil Goes Mining for Big Data." MIT Technology Review, 8 May 2012 (www.technologyreview.com/news/427876/big-oil-goes-mining-for-big-data).
7 Watson, John S. "2012 Annual Stockholders Meeting Remarks." Chevron, 30 May 2012 (www.chevron.com/chevron/speeches/article/05302012_2012AnnualMeetingRemarksbyJohnSWatson9402_356.news).
8 The Digital Oilfield Goes Global" (PDF). Chevron Next*, No. 5, September 2012 (www.chevron.com/documents/pdf/NextIssue5.pdf).
9 Leeber. See 6.
10 Surowiecki, James. The Wisdom of Crowds: Why the Many Are Smarter Than the Few. Abacus, 2005.
11 Arrow, Kenneth. "The Promise of Prediction Markets." Science, Vol. 877, No. 878, 16 May 2008.
ABOUT THE AUTHOR
Jim Plamondon has helped accelerate the adoption of state-of-the-art disruptive computing platforms, including OO application frameworks in the 1980s, Windows in the 1990s, .NET in the 2000s, and cloud computing in recent years. His ability to "win in the API Economy" stems from his experience as Microsoft's internal guru of API evangelism in the 1990s, consultant for other firms throughout the 2000s, and most recently as Director of Developer Relations for Rackspace. He has assisted clients in shaping their strategy toward well-defined business objectives, in hiring the right people to execute that strategy, in training staff to achieve the strategy, and in monitoring company progress versus competitors. He can be reached at email@example.com.Introducing the API Economy: A Dialogue
Did you enjoy this article?
Talk to Cutter today about membership, including one-on-one guidance from and interaction with Cutter's experts and with their peers, access to research, webinars, podcasts, white papers and more.Request trial membership