A CIO’s DevOps approach to resolving the agility-stability paradox

A CIO’s DevOps approach to resolving the agility-stability paradox
Real process change requires a break with the past.
By Bud Mathaisel

For decades, CIOs have applied methods and adopted tools that lead to stability. With good reason: When an IT deployment goes badly, businesses lose customers and risk long-term reputational damage. CIOs can lose their jobs. So business units had to live with a rate of business process innovation that was tethered to the plodding pace of IT’s waterfall software development life cycle (SDLC). These IT systems methodologies are rigorous, but slow and linear.

“The requirements change so quickly that if IT’s pace is in weeks and months, you end up not delivering anything, because the problem changed by the time you deployed your solution,” says Ken Venner, CIO of aerospace company SpaceX.

Speed versus stability is a paradox for IT. Pushing for both at the same time seems contradictory, but the IT organization must strive to achieve this goal if the enterprise is ultimately to become more antifragile—that is, less vulnerable to catastrophe.

Speed versus stability is a paradox for IT. Pushing for both at the same time seems contradictory, but the IT organization must strive to achieve this goal if the enterprise is ultimately to become more antifragile—that is, less vulnerable to catastrophe. (See the article, “The evolution from lean and agile to antifragile,” on page 06 for a further explanation of antifragility and its business relevance.)

Is it possible for the CIO to accommodate a fast-changing business environment while ensuring stability in IT deliverables? It is, but it requires breaking with the safe yet rigid workflow principles of the past. Some companies will be more comfortable with this break than others.

“We have to develop software in other than a Taylorian mindset, where we handed off detailed specifications to a software team to deploy,” says Jez Humble, a principal at ThoughtWorks Studios, a software development consulting firm.

IT can transform itself by adopting agility without giving up stability, but CIOs must change the IT mindset about how to achieve stability. As explored in this article, the key to this change is to adopt a DevOps philosophy that emphasizes speed and efficiency without sacrificing adequate stability. Specifically, CIOs can achieve IT agility and stability by making change a frequently recurring process, which they can accomplish through the adoption of appropriate organic, evolutionary practices used by antifragile, web-scale companies.

The IT agility opportunity through DevOps

Historically, CIOs have trained their organizations and their users that volatility is a threat to stability. IT has responded by adopting Information Technology Infrastructure Library (ITIL) and IT service management (ITSM) standards and practices originally formulated by the British government for technology contractors, similar to the US Department of Defense’s creation of the waterfall methodology. These standard practices made sense at the time, when the world moved at a slower pace. Quality outcomes resulted only from establishing rigor and cross-validation beginning with specifications, followed by disciplined stages of design, development, testing, and release. The big downside: lead times were extended to accommodate this discipline, and that pace does not match today’s business dynamics.

CIOs could strive to balance the emphasis on stability inherent in ITSM and ITIL with a DevOps philosophy that emphasizes speed and efficiency. DevOps enables a closer working arrangement between developers and operations personnel; its best practitioners automate where possible and in the process they make developers responsible for ensuring their code functions well in an operational environment. (See Figure 1 for a summary of the main characteristics of the DevOps philosophy.)

Figure 1
Main characteristics of the DevOps philosophy

Figure 1: Main characteristics of the DevOps philosophy 

Tools that further DevOps and continuous delivery goals help developers gain visibility into the operations environment, streamline their workflow, and automate a number of build, release, and deployment steps. (See the article, “Making DevOps and continuous delivery a reality,” on page 26 for more on related tools.) That visibility is consistent with better controls. (See Figure 2.)

Figure 2
DevOps tool categories within the context of IT service management tools

Figure 2: DevOps tool categories within the context of IT service management tools 

There is no insurmountable inconsistency between ITIL/ITSM and agile methodologies/DevOps. (See the figure on page 13.) Agile methods add flexibility to accommodate variability, improve quality, and speed up the cycle. By using the tools, processes, and techniques described in the business and technical articles of this issue of the Technology Forecast, enterprise IT can change into an organization that offers strategic assets to support innovation, bring new IT products to market more quickly, and respond to dynamic business opportunities—all characteristics of antifragile enterprises.

“For enterprise IT, the whole environment has changed,” Humble says. “Not just in the tool chain, but also in customer expectations of a high-touch experience.”

Agile development, continuous delivery, and DevOps all require organizations to break down silos within the business, including the systems analysts, the software developers, the testers, and the release and operations teams. New features become available as soon as they are ready, allowing instant interaction and iteration, something waterfall processes allow only at the end of the release cycle.

The changing picture of risk in a DevOps environment

Today’s digitized business environments require constant changes to stay aligned with business dynamics. Gone are the days when expensive employees were able to mediate market disruptions and minimize their impact on the business. Many of these roles in sales, marketing, customer support, and even product development have been largely replaced through digital conversations that ride on digital infrastructure. Customers, channels, and partners can’t wait years for new versions of software. The market now penalizes enterprises for lack of speed to market much more than it does for minor software defects.

The challenge for IT is that the traditional approach to IT risk itself introduces delays and is overwhelmingly biased toward managing only small coding errors rather than the risk that IT might fail to deliver business advantage. Traditional IT risk management should be expanded to include technical debt and business model alignment—what PwC calls total IT risk.

If organizations move toward a constant feedback and response cycle in their IT environments with the help of DevOps methods, they can actually reduce total IT risk. The risk curve is not directly proportional to speed to execution in DevOps. The level of total IT risk drops as the commitment and ability to embrace DevOps methods grows.

Enterprises should debunk the commonly held IT department wisdom when it comes to risk, and reassess risk that factors in both the strengths and weaknesses of cloud infrastructure and highly collaborative software development. Jez Humble described what he considers to be the true open source risk picture this way: “Open source actually provides a great way to mitigate some risks because you have systems, packages, libraries, and software where the source code is freely available. Many eyeballs have been on that code. You have a large community of people who have the knowledge to use those systems and support them over time.”

In essence, large-scale collaborative development on its own can change the risk picture entirely, and enterprises would do well to take a fresh look at how cloud-based IT collaboration and automation affects both the magnitude and source of risk.

The table below describes some other major risk categories and compares and contrasts traditional IT risk management with the DevOps/ continuous delivery approach.

Type of risk Size of
impact on
Traditional IT
risk aproach
Dev/Ops continuous
Small coding errors Small Controls through
infrequent code
changes, user testing
Controls through
automated testing of full
software life cycle
Major coding
errors and/or poor
software acceptance
Large Controls through
slow, exhaustive
Controls through automated,
graduated introduction of new
software to user base with
iterative feedback and revision
Business model
shift reduces
Large None Introduces partial solutions
and rapidly adjusts with
frequent code releases
Technical debt
caused by
infrequent updates
Large Requests large
budget for upgrades
Addresses technical debt
routinely within budget

ITIL and ITSM help solve the stability problem. Agility, continuous delivery, and DevOps help solve the IT speed problem. Achieving stable IT operations is a long-standing goal for both, with some major differences:

  • A fundamentally different theory drives stability: As originally envisioned, ITIL and ITSM controlled the process of change and ensured stability by reducing the frequency of change and triggering an IT process if change was introduced. DevOps sees change and evolution in a different light, as a way to boost the enterprise immune system—a key to antifragility. A DevOps and continuous delivery philosophy also views gestation periods leading to big bang deployments as extremely high risk. The longer the gestation, the more likely the business model won’t work, even if the code does. Business requirements change too quickly for a multiyear development plan to succeed. DevOps and continuous delivery advocate smaller, viable deliverables that attempt to address smaller parts of the requirement piece by piece. The operating principle behind DevOps is to make the system amenable to many changes and less likely to have a massive failure.
  • The rhythm shifts from an IT view of how IT changes can be introduced to a business view of changes necessary to compete: This view has been a fundamental gap in alignment between IT and business. Business units become frustrated that their internal IT team is slow to respond, expensive, and not error free. They don’t want to run their own IT, but they conclude they have no choice. IT is frustrated that business units go off on their own, possibly introducing a heterogeneous mix of components and apps that won’t work over time. Startup companies have been the most demonstrable examples of DevOps, but even older businesses can adopt these capabilities and may be in danger if they don’t. (See the sidebar “How two companies move toward the new model.”)
  • The rhythm of change is possible only with high levels of automation: IT must place a tremendous emphasis on design for automation—especially for testing, but also for configuration management, release management, infrastructure provisioning, and other delivery steps. Technical staff should devote quality time to devising and deploying the automation capabilities. Automation will make an important difference in continuous delivery and DevOps proficiency and productivity.

The waterfall approach is still de facto in several industries that have extreme business sensitivity to privacy, data security, independence, or process transparency reinforced by regulations. Financial services must follow strict regulatory compliance rules, reinforced in their public financial accounting (for example, FINRA). Healthcare, nuclear utilities, defense, and other industries also have such constraints, and not every CIO has the opportunity to help transform an entire industry from the outset. However, CIOs in any industry could apply continuous delivery at least somewhere in IT.

There might be an opportunity for agility to gain a foothold in a subsegment of the business, such as a new division, a spinoff, some internal IT (such as HR), or a new acquisition. CIOs can then bridge from that subsegment to more segments of the enterprise after proving the concept and knowing its limitations.

PwC advisory director Keith Katz describes an online equities broker-dealer that started its agility journey by establishing the pair-programming equivalent of co-creation of specifications and code. (See the sidebar “How two companies move toward the new model.”) It doesn’t require an entire industry or an entire company transformation to set the stage.

CIOs must put aside the notion that perfect software specifications are even possible with more laborious methods. Perfect specifications may be an illusion, because IT assumptions of what is needed may not match what the users want. With the agile development approach that DevOps and continuous delivery expand, IT does not design for one point in time for all time. Illusory precision from the waterfall method often didn’t work, either. There is no single deliverable, date, and budget. Continuous delivery is, by nature, iterative. Conversations with the customer will be ongoing, and deliverables will be multiple by design.

The CIO action plan for agility

Deploying code daily is a jarring contradiction to everything seasoned IT professionals believe about stability. Fundamental changes are needed in people, processes, and technology.

Deploying code daily is a jarring contradiction to everything seasoned IT professionals believe about stability. Fundamental changes are needed in people, processes, and technology. But those fundamental changes do realize benefits, as underscored in a recent Rebel Labs survey of 600 IT professionals, which contrasted the average workweek of traditional IT operations teams with the DevOps equivalent. (See Figure 3.)

Figure 3
Overall workweek for DevOps vs. traditional IT operations

Figure 3: Overall workweek for DevOps vs. traditional IT operations 

The survey research confirms that more time spent on automation setup and general testing pays off in the reduced need for support, infrastructure management, and communications time. PwC recommends that CIOs implement the fundamental changes necessary for a DevOps environment.

Step 1: Address the IT skills gap

Competence with new tools and methods must be resolved early. “Technical competence is a major constraint,” says Charles Oppenheimer, founder of Prizzm. Agility is a new area for which most organizations have renegade talent but not organized competence. Some of the talent may be within enterprise IT or in shadow IT groups. Enthusiasts who already have the passion and some experience can form the basis of the team. Organizations also may need to contract some talent from outside. A banking industry CIO interviewed for this article is building on the substantial open source expertise within the bank, while bringing in outside mobility experience. “We may need someone to jump-start us,” he says.

Start by setting up a group in enterprise IT entirely focused on agility. Assign a leader who embraces agility and who grasps the concepts, opportunities, tools, and methods. Ancestry.com, for example, brought an engineering mindset about productivity to IT development, IT operations, and the business. “My group is called engineering productivity, and that name came about because the focus of my group is to improve the productivity and the ability for the business to deliver value to the customer more rapidly,” says John Esser, director of engineering productivity and agile development at Ancestry.com.

Some organizations, such as real estate company iProperty, have cooperated with local universities to recruit talent and train existing talent. Other organizations have found success using Facebook or other social media to recruit. SpaceX’s Venner has brought in recently college-trained IT professionals. “We find talent from a variety of different backgrounds and industries. Some of the talent I’m finding tends to be fresh out of college. They’ve generally been raised in this more agile development environment; they have a stronger orientation to open source tools and not as much focus on the waterfall-oriented development strategy.”

Enterprises may need to restructure their pay and corporate culture to attract and retain the necessary technical talent. Work with HR to assess and address this reality. In addition, resources should be assigned—not “planned” in the traditional sense of SDLC.

Step 2: Establish an agile and DevOps blueprint

Agility can’t be entirely without direction, so CIO leadership is needed to provide clarity on design goals, priorities, and resource assignments. The CIO needs a blueprint to ensure that the overall direction is consistent with the enterprise needs and resources.

Engage the business unit CXOs in a survey to identify the top five business opportunities for agility. The best candidates are scenarios where the dialogue with customers frequently changes and where the IT organization will need to iterate rapidly for new product ideas, markets, and customers.

Some initiatives already may be underway, set up independently from enterprise IT, which can be coalesced into the official road map. The CIO should not view this scenario as an opportunity to commandeer those initiatives, but instead should embrace them as pilot projects that are included in the opportunity road map and that have a goal of shared organizational learning.

Focus on the two-way conversations with the customer, or the potential opportunities for having these conversations, and on conversations that have a dynamic nature that warrants agility. Include IT suppliers, if they can add value, and by all means engage with businesses partners, especially suppliers, distributors, retailers, or staff in marketing, sales, and advertising.

Step 3: Invest in technical tools, software management approaches, and infrastructure

The CIO must determine what technology investments are required to get started. Early priorities include the following:

  • Choose a software management, version control, and change management and programming approach (such as pair programming).
  • Choose a social collaboration tool.
  • Automate everything.
  • Establish the rhythm of continuous delivery and DevOps.

An early step for the agile IT team is to determine how to manage each experiment in the road map. Change management helps ensure that the user community is informed of the change and the rationale. Release management ensures that all testing is completed satisfactorily and that new code will not conflict with something else in code, security systems, or infrastructure. Change management and release management will help ensure that design inconsistencies are caught before the code is promoted to production.

Despite the flexibility required for agility and for the mindset associated with experimentation, agility does not translate into chaos if CIOs provide proper oversight.

Despite the flexibility required for agility and for the mindset associated with experimentation, agility does not translate into chaos if CIOs provide proper oversight. Most importantly, agility does not mean the abolition of change management and release management disciplines. Agility-experienced IT groups advise that rigorous change management and release management disciplines should be in place. When press-worthy major outages occur in the DevOps world, they are traced to gaps in change or release management, and are not attributed to agile IT.

Nationwide Insurance began its agility journey by establishing change management for systems and for the organization, so roles and processes would be clear as solutions were deployed. Nationwide’s change management was a companion to the stepwise evolution of DevOps. Four years ago, Nationwide adopted a monthly release management schedule. “Before that, each of 26 business units had their own release schedule, which was chaotic for users,” says Vijay Gopal, vice president and enterprise chief architect at Nationwide.

How two companies move toward the new model

Case study

Several companies provide examples of IT transformation to the new IT solution delivery model. At SpaceX, an aerospace company, CIO Ken Venner has taken advantage of the momentum from an inspirational CEO and a new business strategy for aerospace, although aerospace itself has high regulatory and self—imposed standards. And within the traditional real estate business, CIO Andy Kelk used DevOps to meet the challenge of enabling a new business model for iProperty Group.


Perfect outcomes and the desire for stability are embedded in the DNA of the aerospace industry. Every aerospace engineer learns at the start of his or her training that a perfect outcome is mandatory. Aircraft and space vehicles should never fail, and that mantra continues through all design, production, testing, and operational processes. IT takes its cue accordingly, and most aerospace IT takes a traditional approach.

Ken Venner, CIO of SpaceX, has an IT background with large companies and years of experience with traditional approaches to major systems deployment. When he joined SpaceX in 2011, he found the opportunity to adopt continuous development methods. The CEO, Elon Musk, wants to create a new approach for aerospace, driving the company and inspiring continuous development.

“Perhaps that is because Elon comes from more of a high—tech background than an aerospace background,” Venner says. “IT releases code at least once a week, and we’re trying to move toward a continuous build and integration environment that uses test—driven development as we’re rebuilding the application suite that runs the company.”

Given a recent history of success, Venner and SpaceX have demonstrated that IT can effectively leverage its new IT delivery model for the business as it evolves to a new model for space launches.


Another example is iProperty, which operates in an industry that has a very long set of traditions. Recently, iProperty chose to launch an entirely new area of business—commercial real estate—in a few months.

The IT development group was accustomed to “a very traditional approach in which IT code was turned over to QA, which would find all the bugs and then pass it to operations, which would run it and not touch it again until there’s a change request,” says CIO Andy Kelk.

That approach would not work in the short time window required to get iProperty into the commercial business. It was a challenge to “get the developers in the mindset that they are responsible for their code from the moment they start writing until it’s in production,” Kelk says. The successful application of agility and DevOps capabilities allowed the business to succeed. The developers saw the tangible benefits early and could make quick adjustments to software as iProperty deployed and then adjusted its business model.

As a result, “We’re able to deploy changes at any time of day—even multiple times a day—with very minimal risk,” Kelk says. For example, he adds, IT quickly responded to a change from a subscription model, which had been the foundation of iProperty’s residential real estate business, to a different revenue model for commercial property.


Releases weren’t routine; each was unique and involved its own dependencies. IT was confusing users, and the organization needed to invest significantly in training each time. Moving to a monthly release schedule was a first step in Nationwide’s evolution toward agility, but an important and symbolic one. It demonstrated the consensus of the 26 business units that they had to trade some autonomy for a better, long-term result.

The infrastructure must be agile ready at the outset. Esser of Ancestry.com says CIOs should start on the technical dimensions by “asking themselves if the IT infrastructure can be leveraged, if it is an asset, or if the business believes it’s a liability.” Given the readily available cloud infrastructure for hire, answering those questions may be one of the easier tasks to accomplish quickly. “We’ve probably seen the largest gains on the infrastructure side,” Esser says. “Previously, even replicating a given server’s configuration was very challenging, if not impossible, and obviously that added a lot of time into the equation.”

Much of the automation required for agility assumes a private, public, or hybrid cloud architecture, and enterprise IT must provision it quickly. One way is for IT to act as a service broker. Phil Berman, a director in PwC’s infrastructure and operations practice, suggests that instead of setting up your own cloud infrastructure, it may be more practical and time efficient to go to a cloud provider (such as Amazon or Google) and acquire cloud services on behalf of the business, rather than the business negotiating for services on its own. “IT still keeps a level of control and governance around all IT, but basically resells Amazon or Google services back to the internal users.”

Berman describes a situation in which an enterprise IT organization entered into a “white label arrangement” with Amazon and Google, and through them provides cloud-computing services to the internal organization. Enterprise IT chooses the preferred providers, consolidates the purchase for lower costs, maintains the service catalog and governance over security, and charges the users for the cloud services and their value add.

Step 4: Automate

Currently, no tool automates everything. But automating various processes will improve agility. For example, reviewable scripts and logs can be created through automation. Then, if something goes wrong, the automation associated with that error can be fixed.

System setup and configuration can be automated (using Chef or Puppet, for example), but testing the setup is crucial. Automate code generation and integration, but test that, too. Add scripts that combine the code generation and integration. Test that, and so forth.

The fast-paced world of continuous delivery requires extensive pretesting, but techniques such as limited release also work well. In the limited release technique, code is rolled out to a sample of the production environment. If it does not meet expectations, the code is rolled back. If it meets expectations and proves stable, the code can be promoted to the enterprise universe.

Venner calls this method test-driven development. Some organizations might encourage different teams to experiment with different tools, especially at the outset. But it is also important to avoid the usual tendency to do extensive research before choosing just one tool and one approach.

Step 5: Remove barriers

The CIO’s job is to enable others to succeed. Unlike the CIO’s role in major SDLC projects, the CIO of a DevOps environment will not manage (some might say micromanage) the details of process work, design, development, or production. Instead, Venner says, “It’s finding, understanding, and removing blockers for the development team, which means having conversations around what obstacles they have—either in terms of the business and the business interaction, or the technology or some process that’s stopping them from getting the jobs done.”

Often, as many as 5, 10, 15, or 20 development teams will be working at the same time. The CIO can’t manage the details or the fast pace of everything, yet he or she will be responsible for the outcome.

The CIO alone can identify and resolve constraints. The transformation from a more traditional IT approach to a more agile-centric approach, the core being continuous delivery, will involve cultural and organizational changes.

The CIO alone can identify and resolve constraints such as infrastructure limits, preexisting contracts (outsourcing commitments or software licenses), business process traditions, organizational culture and behaviors, or lack of training. The transformation from a more traditional IT approach to a more agile-centric approach, the core being continuous delivery, will involve cultural and organizational changes.

The CIO’s horizon must encompass suppliers, channels, and customers. IT teams need the freedom to achieve agility and variability for the customer-facing systems (systems of engagement) while not jeopardizing the core legacy systems. Sometimes that balance isn’t clear, so the CIO must intervene with flexible judgment—agility in an executive sense. The technical tools and techniques may be new, but the CIO’s proven skills as a leader will help the organization succeed even in the face of many challenges.

Step 6: Integrate and modernize legacy systems

Most enterprise IT organizations have legacy software and data systems in place, such as enterprise resource planning (ERP), customer relationship management (CRM), and other systems of record. These systems run well after years of investment and tuning, and they are the core operations of the company. Eventually, CIOs will need to determine how to integrate the agile systems with the core and perhaps modernize the core itself.

Customer-facing systems, the systems of engagement, are an example. These systems may source data from the legacy core or create data that must synchronize with the core. In mobile banking systems, for example, taking a photo of a check with a smartphone to make a bank deposit requires an interface with the banking transaction and clearing systems.

Many legacy systems have significant residual sums on the organization’s balance sheets, and years of depreciation and life remain. These systems are a form of sunk cost, requiring a cost-benefit rationale for replacement. For some industries, the core systems may be subject to regulatory oversight, and for public companies, they are subject to extensive audit reviews.

Not all IT is amenable to agile approaches at the outset. A plan for modernizing the legacy systems will require a situation-specific analysis to disassemble the core and replace modules in a logical pattern. The tools and techniques for the legacy systems are the same as for new systems. To follow the principles of DevOps, a plan for legacy modernization must avoid long delivery cycles and big bang approaches.

Nationwide’s legacy modernization, for example, tackles the noncritical parts of the business first, proving concepts and adapting the user base. These systems are on a cloud, virtualized environment and have a fully secured interface back to the core. This approach allows “gearing” between the agile stack and the core through edge databases and edge applications. Nationwide’s design philosophy for replacement is to make the systems more intuitive, helping to ease the transition for the user base.

For those industries where the core is not readily amenable to agility, the CIO might consider spinning off the legacy into its own environment. Then wrap it with Java and application programming interfaces (APIs) so it can interface with the emerging new environment, and eventually phase it out.

The specifics of the blueprint—tools and methodologies—will vary depending on the new agile applications and the specifics of the core. The great news is that the building blocks are available. In fact, the IT organization may already have experience with many building blocks individually, but not in a holistic, agile way. The CIO has the opportunity to master these valuable capabilities to help transform the enterprise. The CIO should demonstrate agile IT leadership and then graduate to the next level.

Beyond IT: The CIO as agility leader for the business

To become more antifragile, a business will need agility, but it might lack experience in how to become agile. Sometimes the business is stuck because it applies its traditional approaches to change. Or, sometimes, business leaders think they are stuck because they operate in a nonagile IT environment. Either way, IT can prove itself capable in agile IT and serve as a role model for the enterprise.

Customer-facing IT services are the most strategic and are a logical starting point to help transform business processes. Customers expect a rapidly evolving relationship with your business that recognizes their needs and preferences.

Customer-facing IT services are the most strategic and are a logical starting point to help transform business processes. Customers expect a rapidly evolving relationship with your business that recognizes their needs and preferences.

IT has the potential to accomplish this transformation; now it needs to demonstrate its capability. “As we leverage more agile methods, tools, and processes in our [IT] group, we’re getting to the point where I feel the business has a good foundation so we can actually practice what I call business agility,” says Ancestry.com’s Esser, who learned that business process reengineering disciplines helped tremendously in applying agility concepts to the business.

The goal is to add strategic value to a dynamic marketplace. The method of choice builds on agile development and cloud computing with DevOps-centric approaches as well as ongoing evolution and delivery. Business agility will increase as IT agility does.

IT has the tools, methods, and infrastructure to adopt agility now. The CIO must be the agile leader for IT and for the business, because no one else has the toolkits or talents for this task. CIOs need to address the speed of business versus the speed of enterprise IT. CIOs also must lead the modernization of old thinking about stable IT systems. Certainly, no one else can harness the capabilities of the core IT legacy software and data—the systems of record.

Don’t force your internal business partners to go outside the organization or to watch as their competitive landscape is threatened. CIOs have the keys to agility and the unique leadership skills to demonstrate their value to the enterprise, both in IT and in the business overall.