How established enterprises are moving toward DevOps

Vijay Gopal

Vijay Gopal is a vice president and enterprise chief architect at Nationwide Insurance.

Vijay Gopal of Nationwide Insurance discusses DevOps within the context of an insurance company’s view of risk.

Interview conducted by Alan Morrison, Bo Parker, Bud Mathaisel, and Robert L. Scheier

PwC: What is the development philosophy at Nationwide, and how is it evolving?

VG: The biggest characteristic of our development philosophy is that we embrace the agile methodology. At our Application Development Center, we have very good predictability and quality, and we hope to harvest some of those benefits and get more of the organization to embrace the agile methodology.

Our stated goal is to have 50 percent of all our development go through the Application Development Center by the end of the year. The processes of test-driven development and involving the business as part of the development are both catching on and proving to be very successful.

We perform more than $400 million worth of build work every year

But when it comes to the cloud-based development paradigm, it’s a little harder to gain a foothold. Our organization is just getting their heads wrapped around agile. We perform more than $400 million worth of build work every year, so we must be able to do that development at scale. We are one of the few CMMI [Capability Maturity Model Integration] level three certified shops that also embraces agile practices, and we also must be mindful of organizational change management as it relates to how these systems are being deployed to our user groups.

We see some challenges in embracing the cloud-based development. That said, we are experimenting with it for our internal applications. We have an area called enterprise applications, and we are developing an application catalog. To build this catalog, our developers take the functions that associates frequently access from our intranet portal, and then our developers create easily accessible widgets. Instead of going through five or six clicks to find out how much time off they have, for example, associates could click on a widget and get that information quickly.

We are becoming better informed about what it takes to do that as a company, while still getting the organization to shift and embrace the agile principles more effectively.

PwC: Are you able to do more frequent releases?

VG: We’ve had a monthly release calendar for the past four years. Before that, each of 26 business units had their own release schedule, which was chaotic for users. Going to a monthly release schedule provided some discipline for managing all the defects that get introduced because of change. We want to take a closer look at how companies such as Yammer or Salesforce.com do daily or weekly releases while keeping the user experience from being disrupted.

We’ve had a monthly release calendar for the past four years

We need to have some discipline when we roll change out to the call center associates or the agencies, for example. When we introduce change into these environments, we must be very careful about the impact it will have on people who have been through a training program, have used some of the features for a long period of time, and are accustomed to an approach where the field associates deliver some training to orient them to those upcoming changes.

It’s the other side of the spectrum from the world of the web companies, where code once validated gets released. Those applications are intuitive enough that the users don’t need to be trained to take advantage of the features.

The way I see Nationwide moving toward more frequent releases is by learning from all the SaaS [software-as-a-service] solutions that we are starting to embrace, and from the packaged implementations that we’re bringing into our environment more and more.

PwC: How do you deal with legacy systems in a frequent release context?

“We would need to prove this agile development in some of the noncritical areas of our business first, before we could start talking to our business about this kind of an approach.”
 

VG: We have some antiquated systems that have not been developed with the degree of consistency or using the principles that we would like to have. We’re investing significantly to move toward that goal. But we also need to keep in mind that we operate in a regulated industry, and we can’t release code that could compromise customer information or some of the financial and sensitive data that we have. We would need to prove this agile development in some of the noncritical areas of our business first, before we could start talking to our business about this kind of an approach.

PwC: Could you tell us a little bit more about how you’ve been able to bridge the agile approach and the working code approach in a package world?

VG: Even when it comes to legacy modernization, such as when we implement a packaged solution to replace our core policy administration systems, we take an agile approach. We put a heavy emphasis on working code rather than project status reports, for example. From a DevOps standpoint, we are rolling out more and more of the private cloud solutions, especially in a Java environment.

When going the package route, you must make some conscious decisions about how much customization you will take on. Our philosophy has been that the package implementation itself will be successful if you also make some business process changes that help you take advantage of the package implementation.

We get a feel for how the application will behave. And to go along with the package implementation, we also have a business effort called product simplification. Our product varies by state to reflect the differences between how marriage is defined in one state as opposed to another, for example. That’s an opportunity to give more flexibility to our field offices.

That’s a huge change for our business, but the field offices had to sign up for that so the organization could see all the benefits that come with the package application. Now once we have done that, the question becomes one of the release schedule and clarifying whether we’re taking a state-based or a product-based release schedule.

We started with the simplest products first. We make accommodations for the business rules that need to be put in place and any minimum customization required to make that happen. That becomes the first iteration deliverable. When the updates come from the package supplier, these principles will help protect us from being disrupted by an upgrade.

PwC: The package must be designed explicitly to support this change independence, yes?

VG: These packages do. The good vendors have figured this out. You have the clear product functionality—code that’s provided by the supplier. Where we spend a significant amount of time is with the configuration, or business rules. We make those changes, test them, and then we release them.

PwC: When you refer to legacy applications that are less modern, they’re less likely to have these capabilities, yes?

VG: Yes. In the case of business rules, there’s no clear separation between business rules and the rest of the logic, and that makes it harder.

PwC: Can you talk a little bit about where your operations shop is in terms of its ITIL [Information Technology Infrastructure Library] adoption and whether or not you see any tension between agile and ITIL?

VG: We definitely see some tension. ITIL adoption has been mature in a couple of disciplines, such as incident and problem management, and we’ve realized some benefits from adopting these practices. The tension between the agile and ITIL practices is somewhat mitigated by our monthly release calendar. The teams understand that all our iterations in the test center are two-week iterations, and we’ve structured our work within that monthly release calendar.

PwC: Do you think you may move more toward DevOps in the future?

VG: I think we are more likely to go toward DevOps, but before we get there, we need a standardized infrastructure environment. While we’ve embraced virtualization, there’s more of a craftsman mentality in how Java fills the solution. To deploy and manage infrastructure in the scale at which we consume it, we need to rationalize workload patterns with our development teams.

Configurations and automating the management of our solo configurations and our infrastructure are some of the intermediate steps we need to take. We are in the process of taking those steps through a couple of efforts: one is the private cloud, and the other is an effort to simplify the I and O [infrastructure and operations] catalog of services.

PwC: Once you’re comfortable with those, then you’ll start looking to do more of a full life cycle, DevOps-style approach. Is that a good way of thinking about it?

VG: That’s absolutely how we are approaching it. We’ve finally reached a point where we cannot defer the basic hygiene work that we need to do around systems anymore, in terms of the technical debt and taking care of some of the risks we have in our systems. Some of that was deferred in favor of rolling out more business capability.

PwC: Have you been able to explain this concept of technical debt to business leadership?

VG: Yes. A couple of years ago, we consolidated the amount of technical risk we have in the system. The first step was to get visibility about how much risk we had in the system and finish that step in time for our annual planning cycle. A large business transformation committee makes all the decisions on allocating funds for projects.

We highlighted any significant risk, provided an estimate, and proposed that we invest to mitigate the risk. We hoped to avoid a situation in which one business unit spends a significant amount of money because they’ve made a good case while another business unit has much higher risks that are not being tended to because a business capability is being rolled out.

This year we’ve discussed baking into our operational budgets the mitigation of technical risk or taking care of technical debt. When people run budgets, they tend to assume that they can base their budget for next year on the money they got this year. We’re approaching technical debt as an item that will be added to every year, so we assemble the risks, prioritize them, figure out what needs to be fixed, figure out the level of effort required from each of the business solution areas, and explicitly add that amount to their budget. It’s a little more of a disciplined approach for shared risk on platforms that are used by more than one line of business.

We have three broad categories that we bring under technology risk. The first is technical debt, such as a compromise made to meet project delivery deadlines. The second is currency, such as deferring upgrades to an environment because of the cost associated with it. The third is consumption management, or efficiency, where making changes will help you optimize how much users consume. All of those we put down as risk, and we manage them accordingly.