Reconsidering risk in a continuous delivery environment

Jez Humble

Jez Humble is a principal at ThoughtWorks Studios and the author of Continuous Delivery.

Continuous Delivery author Jez Humble says culture, rather than risk, is the real obstacle to transforming IT.

Interview conducted by Alan Morrison, Bo Parker, Bud Mathaisel, and Robert L. Scheier

PwC: Many CIOs are accustomed to buying packaged software or subscribing to services on the web, and they may even outsource the development that must be done. They may not be mentally engaged with the issues you explore in Continuous Delivery. Is there an argument for these CIOs of established enterprises to reengage?

JH: You still often see the same problems with packaged and COTS [commercial off-the-shelf] software, which people buy because they don’t want to do custom software development. They see that as risky.

They think COTS is going to handle a particular business process, and then they find their process does not match what the COTS package does. So they decide they need to customize the COTS. They end up doing custom software development to customize the packages they bought. And packaged software is not normally designed for that.

As a result, they pay a lot of money for a consultancy to heavily customize the software. That’s extremely painful, and upgrading the software also becomes extremely painful. Often, the vendors that sell the packages will not support heavily customized versions of the software. We see this all the time in enterprises.

PwC: Do we need a whole risk reassessment, and do we need to revisit some preconceived notions about where the risk actually lies? A CIO might have a risk concern about open source, for example. Where do you place that risk?

JH: In our experience, enterprises are very aware of that risk. But they don’t seem to be quite so aware about the risk of many of their legacy systems already in place. I’ve given talks and consulted extensively in enterprises. And here is one of the standard questions I ask: How many people are running mission-critical systems on legacy software or on legacy packages, where the vendors no longer support the packages or the hardware they’re running on, and where you don’t have the source code to the customizations?

“People don’t want to touch those legacy systems, because they worry about breaking them. But the more you don’t touch them, the more the organization loses the muscle memory associated with being able to manage those systems over time.”

People don’t want to touch those legacy systems, because they worry about breaking them. But the more you don’t touch them, the more the organization loses the muscle memory associated with being able to manage those systems over time. And then it gets more and more risky.

One of the things I read this past year that has really inspired me is [Nassim Taleb’s] book Antifragile, in which he talks about the behavior of systems and how they respond to unexpected events.

Like financial markets, legacy systems represent a huge risk to organizations. Inasmuch as organizations do need to change those systems eventually, it’s very risky. I think organizations always need to look at how to mitigate that risk by making sure they’re up to date with current software and current trends and what’s supported in the market.

Open source actually provides a great way to mitigate some of those risks, because you have systems, packages, libraries, and software where the source code is freely available and many eyeballs have been on that code. You have a large community of people who have the knowledge to use those systems and support them over time. Companies have problems hiring people to support or maintain the systems that they’re running in production. They don’t have the source code anymore. A lot of these open source code tools have large communities of people. So it’s much easier to hire people with experience, and it’s much easier to maintain the tools because there’s a large community built up around them.

Vendors are going out of business or being acquired at an ever higher rate. For enterprises, that is a much higher risk to the software they’re buying than open source.

PwC: What are the points of friction between the DevOps or continuous delivery approaches and other approaches such as ITIL [Information Technology Infrastructure Library—a collection of best practices for IT service management] that have been around for years?

JH: There doesn’t need to be a conflict. I have the ITIL Foundation certification. I’ve studied the framework extensively, and based on my study, ITIL is compatible with agile and continuous delivery. The main point of friction is often the change management process, but ITIL provides several mechanisms for lightweight change management processes, which are completely compatible with continuous delivery.

ITIL is a library of good practices that should be adapted by the organizations that implement them. But frequently that is not the case on the ground. Some people have come in with a very strict, heavyweight implementation of ITIL and said, “This is what you have to do.” And that’s neither in the spirit of ITIL nor is it the only possible way to implement ITIL. Unfortunately, a lot of companies have implemented very heavyweight practices that actually increase the risk of things such as downtime and lack of stability in production.

PwC: To use Geoffrey Moore’s terms, the development model for systems of engagement is entirely different from the development model for systems of record and the core IT. Systems of engagement are often heavily dependent on the public cloud, which is another set of risks. How do you see systems of engagement in this risk environment that we’re talking about?

JH: I think public cloud has made it a lot easier to respond to the market. Software as a service, infrastructure as a service, and platform as a service—these systems pose risks that are different from traditional enterprise IT risks. Those risks include issues such as security and, in particular, making sure you don’t put data in the cloud that could then be compromised.

There are also architectural issues. Building software for the cloud requires architectural approaches completely different from traditional IT approaches. For example, you need the ability to scale horizontally so you can deal with volatility, such as making sure your software doesn’t go down if one data center goes down—which has happened several times recently.

“People are very nervous about the security of public cloud, but time and time again we see hackers getting into company data centers, including potentially government-sponsored groups.”

People tend to be very nervous about the risks of the new technologies, and they tend to be somewhat blind to the risks posed by their existing platforms. For example, people are very nervous about the security of public cloud, but time and time again we see hackers getting into company data centers, including potentially government-sponsored groups.

It’s very easy to be skeptical of this new thing and be very nervous, but it’s just a different set of risks. It’s not a worse set of risks. The business is going to force these people to get up to speed. A lot of this skepticism is driven by fear and nervousness about change, which is understandable. But that’s not going to be addressed by people putting their heads in the sand.

PwC: The collaborative development cycle that happens on the open web is very powerful. How are enterprises approaching open collaboration? Are there certain kinds of development they can do in the open?

JH: Some companies are starting to do so. I think things inevitably will move in that direction, because it’s the only way to move sufficiently fast to address the business needs.

Companies such as Etsy are driving a competitive advantage from being able to do open collaboration. Etsy is not a small company. The company has put a lot of work into open sourcing a lot of its infrastructure. Facebook is open sourcing a bunch of its stuff.

“We’re in an age where the life of the Fortune 500 company is steadily shrinking. Continuous evolution should be a part of every company culture.”
“The biggest problem is culture.”

Trying to hire people who can build software in this way is really hard, and being able to engage with an open community and prove that you’re contributing back is a really powerful tool in trying to hire in skills.

An article in Forbes a year or two ago described how every company is a software company. That’s the shift I’m observing. How do you actually execute on that? The biggest problem is the Taylorian management paradigm of the 19th century, which is still very prevalent in our industry: You can drive out a very detailed specification and then throw it over the wall to IT. A bunch of worker bees do the boring and trivial work of writing the code, and then they toss that over the wall to operations to run to eternity.

That model of building software is not going to work. It’s impossible to predict in advance how customers will respond. Building a really large, complex specification is not a good way to mitigate the risk that the software may not deliver a return on an investment.

Companies need to understand that. The main problem is hiring the people who know the details of the continuous delivery methodology. They are in short supply, and they’re not going to work in a company that has a traditional software factory approach. Companies need to take a very lean, startup kind of approach to building development teams—try some things and experiment.

PwC: What’s the biggest obstacle to moving to this more experimental approach?

JH: The biggest problem is culture. Organizations need to take a cultural approach in which they experiment, try things out, and manage risks by not investing too much up front and by enabling the team to experiment and innovate. You can be very incremental about growing those skills, and then you can gradually seed them out through your organization and grow the rest of your organization in that way.

The lean startup is something that Intuit has been known to adopt, for example. Intuit is not a small company. There are proven models for adopting these approaches. You need to focus on creating a culture of learning and experimentation and being very incremental in what you roll out.

We’re in an age where the life of the Fortune 500 company is steadily shrinking. Continuous evolution should be a part of every company culture. This is a journey. We’ve taken step one in the journey, and we will need to try some other things as well, as we continue to evolve.