DevOps Metrics to Ensure Your Pilot Measures Up

DevOps ushers in new processes, new teams and new technology, so how does one define success? In this final article in our DevOps adoption series, we’ll share our thoughts on a few concrete DevOps metrics your team can use to measure positive change, and just as importantly, why they matter. As the old adage goes, what gets measured gets done and when it comes to DevOps, the overarching impact we should be making is to Time to Business Value.

What is Business Value?

While this term can be quite amorphous, you will frequently hear it in DevOps and Agile circles, and at Flux7 we define it as helping the business achieve its stated goals and objectives. (For additional background, check out our article on Seven Business-Impacting Reasons to Adopt a DevOps Model.) Underscoring the need to define, capture and share business value with C-level peers, Gartner in its May 2018 report, Targeting Business Buyers of IT Operations Management Software, has found that “by 2021, 80% of all software buying decisions for all enterprises will be made outside the IT organization by nontechnical C-level managers — up from 50% today.”

As the business environment becomes increasingly competitive (it feels like new disrupters enter new markets daily,) DevOps teams need to think not just about delivering business value but doing so as quickly as possible. To do so, service level indicators (SLI), such as error rates, transaction throughput, availability, request latency, etc can be good indicators as they highlight what is important to your customers. As you think about measuring the success of your DevOps pilot, have the COE talk through how you might measure these service level objectives to reach a defined target value or range of values for a given SLI as this can help set expectations for the pilot upfront.

Beyond SLIs, here are a few meaningful technical measurements your organization can use to gain a deeper understanding of your time to business value, broken down into four easily measurable proxy metrics. (Note that the first two are development-focused and the latter two are operations-focused.) Tracking through to business value, these technical metrics enable your business to illustrate how you are developing products faster, expanding customer engagement channels, increasing return on IT assets and optimizing operations which positively impact revenue, brand, customer experience, and engagement. 

1. Provisioning Velocity

   — How quickly can your team spin up new infrastructure, provision applications, and more.DevOps automation should empower your team to build CI/CD pipelines that increase the velocity of services, allowing you to automatically build, test, and deploy code based on defined release processes. The outcome of added automation is often a vast improvement in provisioning velocity.
   For example, in the work we did for a Fortune 500 manufacturer, where we set up infrastructure as code, the Flux7 DevOps consulting team was able to decrease the time for server procurement to mere minutes. And, with new scripts for bootstrapping the company’s MongoDB cluster, we were able to shrink a process that initially took the DB admin days to just 15 minutes.

   Technical metric to business value translation: In addition to releasing new services faster, increasing your provisioning velocity means that your team can more easily test new ideas, fail fast, and more quickly innovate for the business.

2. Velocity of Updates

   — How frequently is your team able to issue service updates.As customers become accustomed to — and expect, even — frequent updates to services, the ability to increase release frequency is critically important. Moreover, if it’s easy to push out feature updates, then it’s something that development will be inclined to do.
   One approach some organizations take to speed update velocity is to replace a monolithic app with microservices. This allows developers to focus more on creating and delivering code for their discrete service. This, combined with a smaller, simpler code base allows developers to quickly and easily update their service.

   Technical metric to business value translation: When development can innovate much more quickly, they increase the velocity of product updates and grows code quality. These benefits drive advanced business value through improved customer experience, deeper customer engagement and potentially a competitive advantage in the market.

3. Mean Time to Recover

   — How quickly can your team recover from a failure; does your team have confidence that it can failover should something go wrong.With a primary goal to deliver services, the ability to quickly recover from failure is critical. With DevOps architecture in the cloud, organizations can quickly and easily failover to a backup site, (often in an alternate AWS Availability Zone). Or, some organizations might choose to use a blue-green methodology in which case they’d fall back to the prior deployment should an error ensue.

   Whether you are failing over to a prior deployment, a backup or disaster recovery site, this can give your team much-needed breathing room to tackle the initial root cause. According to the 2016 State of DevOps Report, high performing DevOps organizations not only recover 24 times faster from failures, but they also have three times lower change failure rates.

   Technical metric to business value translation: With fewer failures and faster recoveries, your team can spend more time creating strategic business value and ensures the highest possible levels of customer satisfaction.

4. Scale on Demand

   — The ability to scale up and down, as needed, on-demand.Scalability ensures efficiency and the ability to support business needs when the business needs it. A core tenet of DevOps is to treat your servers like cattle, not pets. Doing so makes scaling systems up and down effortless, ensuring that they are delivering maximum value at all times.
   For example, we had the opportunity to work with Rent A Center on its e-commerce site, which required the ability to scale up to meet peak demand and back down when fewer resources were needed. In this way, they were delivering business value by ensuring optimal customer experience and by being fiscally responsible. Indeed, over nine million people visited the Rent A Center e-commerce site over Black Friday. Its system scaled seamlessly to manage this 42 percent increase in traffic, all without missing a beat.

   Technical metric to business value translation: Scalability on demand helps increase return on IT assets and optimizes operations. In this way, operations are able to positively impact top-line revenue, help manage brand reputation and ensure a good customer experience.

Create a Yardstick

It’s ideal to have a ‘before and after’ picture of these metrics within your environment. However, we find that in many cases, organizations really have no idea of how long it takes them to provision or update their systems or what their meantime to recover might be. For those areas where you have existing measures, we highly recommend that you document them.

For those areas where you have no measures, we recommend that you collect any data points you can. One way that many organizations approach this is by interviewing staff to get their take. For example, how long do developers think it takes to update a system? Or, what is the operations team’s take in the meantime to recovery? With these findings in hand, the new DevOps team will be able to better paint a picture of the impact of your DevOps pilot, encouraging momentum for future DevOps projects.

In prior articles, we’ve talked about the best approach for selecting a DevOps pilot project, and how to select the best-fit team to staff it. With SLIs and these four proxy metrics for measuring time to business value, your new DevOps team can start illustrating their impact to the business, helping to create an enthusiastic reception for your pilot project. Moreover, your team will have a solid best practice foundation for ongoing measurement as DevOps spreads throughout the enterprise.