An aerospace industry CIO’s move toward DevOps and a test-driven development environment

Ken Venner

Ken Venner is CIO of SpaceX, a space transport company founded in 2002.

Ken Venner of SpaceX explores agile development’s latest evolution.

Interview conducted by Alan Morrison and Bo Parker

PwC: What differences are you seeing in how developer groups in various industries approach more frequent forms of delivery?

KV: Although there are some very unique requirements when launching cargo and people to space, I’d say aerospace in general is a little bit behind other industries. But SpaceX—just based on the fact that our founder, Elon Musk [also co-founder of PayPal and Tesla], comes from more of a high-tech background than an aerospace background—inspires a continuous development effort. My development team is aware of how the build process takes place and how test-driven development is evolving at companies like Facebook, LinkedIn, and Google. Not all of it is applicable when dealing with spaceflight, but our goal is to take best practices from multiple industries to create the best possible solutions for SpaceX. Right now we release at least once a week, sometimes midweek, and we’re trying to move toward a continuous build and integration environment that uses test-driven development as we’re rebuilding the application suite that runs the company.

We release at least once a week, sometimes midweek

PwC: Are you using this build environment for your core systems, for systems of engagement, or both?

KV: With our ERP [enterprise resource planning] platform and financials package—systems of record—we’re adopting a strategy of weekly release, desired zero downtime, and continuous build, where we deploy capabilities when capabilities are defined to be ready. The ERP platform uses a custom build application internal to SpaceX. Once a week, we roll out a new release of the application. Longer term, we’re looking at creating a promotion process that supports zero downtime deployment as well as the ability to trial features with targeted user groups, which would allow us to roll out smaller features more frequently.

PwC: What kind of technology and methods are you using?

KV: We’re predominantly in the Microsoft stack, IIS on the front side, and lots of JavaScript. The build servers have integrated testing, and we’re moving toward a test-driven approach. We also use PHP and MySQL as another critical technology stack in which we’re doing a reasonable amount of development.

Overall, we have a modified agile environment, and we’re interacting with the users every day or every other day.

When we need to build larger pieces of functionality, we set up a feature branch and have our users work their way through these new capabilities. We share action models in the business case for how that feature is supposed to work and how it integrates or touches other capabilities. When this feature is thought out well enough, it drops down onto the core trunk and comes out as part of a release in one of the release cycles.

PwC: Why would you want to make such frequent revisions to your core manufacturing and financial applications? We tend to think of those more as systems of record, where requirements don’t change that quickly.

KV: This approach might not make a lot of sense for enterprises that are not trying to dramatically transform or attack a new market segment and need to get new features.

At SpaceX, we are redefining how our industry operates, and therefore we’re creating new and different business processes.

And I work for a boss who is always interested in increasing efficiency and extracting steps from the process. Our company has a philosophy that no matter how well you’re doing, you can always do it better. Therefore, we’re always looking for a way to improve the process—to either increase automation or reduce a non-value-added operational step. To do this, we need to enhance, modify, or change our processes and the supporting systems.

PwC: Do you have some very specific requirements that might make a difference in whether—or to what extent and how—you adopt open source, as compared to a consumer photo-sharing site, for example?

KV: Well, embracing open source for internal-only consumption doesn’t have the liabilities or limitations that it would if it were the foundation of the product you’re bringing to market. Enterprises should be more willing to look at open source for internal consumption as an option and a solution, because they don’t really have an IT issue if they’re using it only for inside consumption.

PwC: In the past, you’ve mentioned a talent shortage. Can you talk a little bit more about where you find your talent, how you manage it, and the nature of the supply of the capabilities?

"We find talent from a variety of different backgrounds and industries. Some of the talent I’m finding tends to be fresh out of college. They’ve generally been raised in this more agile development environment; they have a stronger orientation to open source tools.”

KV: We find talent from a variety of different backgrounds and industries. Some of the talent I’m finding tends to be fresh out of college. They’ve generally been raised in this more agile development environment; they have a stronger orientation to open source tools and not as much focus on the waterfall-oriented development strategy.

We can compete with companies such as Google, LinkedIn, and Facebook for developers, because our goal is to go to Mars. We have an intriguing mission that attracts top-end talent. They’re aligned with the mission of the organization and want to do something interesting and exciting.

PwC: What’s your approach to management and governance at SpaceX?

KV: It’s finding, understanding, and removing blockers for the development team, which means having conversations around what obstacles they have—either in terms of the business and the business interaction, or the technology or some process that’s stopping them from getting the jobs done.

At a higher level, it’s ensuring that the direction of all the development teams aligns with some larger vision. So that involves pulling up the team and making sure that as they finish building features, the features are taking the company in a particular direction and achieving a goal. I’m interacting with my peers to make sure the teams deliver—within the time frame—the features that are designed to win the business. And then we’re making sure we have the appropriate resources, and we’re managing the expectations of the organization.

PwC: Related to those goals, how would you describe SpaceX’s business model?

KV: At SpaceX, we use smarter engineering to create highly reliable rockets and spacecraft at a reduced cost. We believe our approach will help reduce the cost of launch down so low that there’ll be new use case models for it, and the features we will need so we can manage the number of people attempting to do launches will shift over time.

PwC: What process have you been through personally that led you to understand the potential of this agile, DevOps-oriented approach?

KV: Well, I’ve always liked this area and have tried as hard as possible to focus on this leaner, more aggressive aligned feature set versus the big cascading waterfall model where you’re talking in years to get something done.

The requirements change so quickly that if IT’s pace is in weeks and months, you end up not delivering anything, because the problem changed by the time you deployed your solution. The business moves quickly here, and so one key challenge is investing enough time to understand how the technology stack works, how to use it, and how to change a thought process.

“Once you start getting your arms around agile development, it’s really not that hard to think of a better way to do things. You just need to let go of some of the old paradigms you had and embrace newer structures and newer ways of getting things done.”
 

Once you start getting your arms around agile development, it’s really not that hard to think of a better way to do things. You just need to let go of some of the old paradigms you had and embrace newer structures and newer ways of getting things done.

In general, though, the predominant approach is, “Let’s get into this thing and just start working it. Every day, we’ll figure out what piece we’re developing, whether to play it up or make it visible or wire it up underneath. And we’ll figure out if we’re trending in the right direction or not.”

PwC: How has the advent of the cloud changed the way development gets done?

KV: Virtualization lets you spin up a large number of instances of your application environment. Therefore, the ability to have multiple parallel scrums going at any one time is relatively cost-effective and relatively simple to do nowadays.

In the old days when you had one development, one test, and one production person, creating code that messed up the world was relatively difficult to do. Today in the world of virtualization the attitude is, “I’ll just spin up another VM [virtual machine] and get another environment going.”

PwC: What advice would you give CIOs in other industries who are considering a more test-driven, frequent-release-oriented development environment?

KV: There are different approaches for different points in time of an application life cycle. This agile methodology makes a lot of sense in an application that’s either new in its life cycle or in the process of a major redesign.

For bringing new capabilities to an existing application or changing a business process, agile development is the fastest way to keep all players aligned, focused, and working on the same thing without having tons of delay. Users generally are not trying to talk to you about a problem they have today that you say you’re going to solve in a month. They have the problem right now, they need a solution right now, and they want to know what you will do for them right now. By using this more aggressive strategy, you basically get people to spend less time iterating on all the possible alternatives. They’ll invest their energy in the direction they’re heading in.