CodePipeline Pipeline Execution History and DevOps Metrics

Building a continuous integration and continuous delivery pipeline is a goal of many enterprises as they look to increase their agility and speed time to market. More to the point, as part of a healthy DevOps environment, CI/CD pipelines deliver business value through automation that grows developer output and strategic contributions. For AWS-based DevOps environments, many enterprises look to AWS CodePipeline to help facilitate their CI/CD as it integrates easily with other AWS services — and a broad set of ecosystem tools — and provides a consistent set of quality checks for code.


We are pleased to note that with a recent update, the CodePipeline team addressed what had become one of the service’s most requested features: a view into the history of your CodePipeline pipeline executions. This relatively new feature enables you to now easily manage and understand your day-to-day software release workflow by giving you visibility into the history of your pipeline activities.

DevOps Metrics and Pipeline History

The history of your pipeline execution activities provides important information and metrics, such as what is the average build time for the pipeline, how many builds succeeded in a given period of time, etc. Insights such as these are important for audit purposes and for measuring DevOps success — as well as for identifying ways you can improve and extend your Devops initiative — the continuous improvement loop.

Some of the most commonly recommended DevOps metrics that can now be gathered from your AWS CodePipeline pipeline execution history include:

1. Deployment frequency
   This measurement of how often and how quickly new features are rolled out can tell you a fair amount about the effectiveness of your process. For example, are there any bottlenecks in the process and if so, are these steps even necessary and/or are they in the right order to keep the workflow moving smoothly? Are there upstream issues with the team itself and its responsiveness to new feature requests? And, do they have all the tools they need at their disposal to ensure collaboration happens effectively?
   
   Ideally, deployment frequency as a metric should remain flat or increase over time. Any decrease or negative bump should be examined for process and team-related bottlenecks.
2. Percentage of failed deployments
   While this metric is self-descriptive, it is a key measurement of success as DevOps depends on repeatedly delivering new code in a predictable, uneventful way — similar to a drum or heart beat. Quickly turning out failing code is counter-productive to creating value for the organization. A high failure rate (or one that does not decrease over time) often points to an issue in the DevOps process. Key questions your team should ask to suss out the issue include, what percent of deployments have caused outages? What percent have caused customer support issues? If your company uses other measures of deployment quality, they should be applied here as well.
   
   This metric should decrease over time as a team’s experience and skills in delivering code should increase, translating into fewer errors and failures.
3. Build count by day
   While deployment frequency may indicate the speed and consistency of your overall feature releases, daily build counts will tell you how many new lines of code are being deployed, suggesting the complexity of your code as well.
   
   This metric should be consistent and should have a corollary to the cadence of your team’s sprints. Any negative spikes in build count should be assessed right away. The questions above are a good place to start to investigate any issues.
4. Build duration
   This metric examines how long it takes to complete a build and helps you gain an understanding of how quickly your app is growing. While code builds can be the most time intensive part of the CI/CD process, they are important to watch carefully. Another angle to examine here is how build times vary between different environments. At the end of the day, if builds take too long, it will dry out your pipeline, keeping it from flowing smoothly.
   
   Like other measurements of your code pipeline, you should expect consistency in build duration times, if not a slight improvement over time as your team’s skill set grows. Any disruptions to this consistency should be flagged and investigated.
5. Lead time for changes
   Last but not least is tracking the time it takes for code to make its way from development to production. This metric examines how long it takes from the time a developer begins writing brand new code to when that code is deployed into production. This measurement of the development cycle is a good way to gauge  the efficiency of the DevOps process and supporting systems as well as the capabilities of the development team. Lead times that are too long can tell you if your development process has inefficiencies built in and if there are bottlenecks in your deployment process.
   
   Lead time for changes should decrease over time as the team becomes more and more familiar with the process.

All these metrics can now be easily collected by accessing AWS CodePipeline pipeline execution history data. (The AWS CodePipeline user guide has detailed steps on how to access your pipeline histories for analysis here.) The interesting thing to note is that all the data that is returned regarding past build statuses are in JSON format which is very helpful for running analytics and conducting any further processing.

As to what details you can analyze, the AWS CodePipeline console provides change details associated with each execution, and allows you to track the status, start time, end time, and duration of every execution, for each of the pipelines associated with your account. In addition, you can track:

• When actions last ran in the pipeline,
• If a transition between stages is enabled/disabled,
• Any actions that failed,
• Revision details associated with source artifacts, such as execution IDs, status, and commit IDs and messages.
• Details for all pipeline executions for which history has been recorded.

Note that execution history is limited to the most recent 12 months.

As enterprises hone their CI/CD pipelines to generate greater business value, keeping an eye on DevOps metrics is important to ensuring processes and systems remain efficient. With a deeper understanding of your day-to-day software release workflow and your development lifecycle, you’ll be able to quickly identify any bottlenecks and flag areas for improvement, helping your business achieve important DevOps model benefits.  If you are exploring how continuous deployment can benefit your organization, check out our resources on the topic or subscribe to our DevOps blog below.

• Setting Up an Efficient Continuous Integration and Continuous Delivery Pipeline
• AWS CodePipeline, CloudFormation & Continuous Delivery
• AWS Developer Tools Provide Continuous Deployment to Amazon ECS
• Building a Continuous Integration Environment in AWS
• Optimized Code Delivery Pipelines