KPIs to measure traditional and agile project delivery

As a project manager (PM), it's easy to get caught up in the day-to-day activities of project delivery and fail to take a step back and measure the work. But by taking a holistic view and quantitatively measuring your effectiveness, you can identify and resolve pain points and improve your delivery during current and future projects.

A key performance indicator (KPI) is a measure of performance. In the context of project execution, KPIs allow project managers to understand where their projects are succeeding and where there’s room for improvement. Having distinct KPIs and metrics to measure a project is essential for effective delivery. Here we’ll discuss common KPIs to consider when running a traditional waterfall delivery model as well as in agile.

Waterfall vs agile

Before diving into the KPIs for delivery, it's important to point out the high-level differences between agile and the traditional waterfall lifecycle. Waterfall is a linear lifecycle divided into distinct phases (design, development, testing), which are completed sequentially. Agile, on the other hand, offers more flexibility with an iterative and incremental approach. Your different software development phases are performed, producing a usable product increment, as the outcome of each sprint. It’s important to take this into consideration when we measure project performance.

Four KPI areas

In any project, the effectiveness of your delivery can be measured by considering four areas: Schedule, Scope, Budget and Quality. This is often called the triple constraint in project management, where the quality of work is tied to a project’s budget, deadline and scope.

Scope in waterfall

Work delivered vs work planned

When discussing scope, one of the first things that comes to mind is what work is being delivered. In a traditional model, the approach is to complete the development of all requirements identified at the planning phase and deliver a final product. Work delivered is a direct measure of completing the scope planned against the scope delivered. This includes any change requests approved throughout the project.

How to measure

• Score/weigh work using a common scale
• Map team capacity to a measure of the work they can complete given their availability
• Planning will yield a breakdown of work to which each team member commits for a given work cycle. The summation of the tasks measured provides a measure of the “work planned”
• The measure of the work delivered is determined when the team is “pencils down” in the project

Oftentimes, complexities associated with delivering a piece of scope will emerge after the team has started its work. New information may reduce the importance of a scope item considering the time/dollars required to complete it, so it's important to take this into consideration in terms of your planned work.

Planned capacity vs realized capacity vs scope delivered

The purpose of tracking capacity is to understand the relationship between the project staffing model of the organization and the ability to achieve staffing needs for its execution goals. As a PM, it's important to understand the project team’s available capacity, prior commitments or competing priorities before the start of any project. By measuring each resource’s capacity and utilization with a uniform metric, the PM can track capacity as work is delivered and new needs arise. For a waterfall model, it's important to plan resource capacity in the planning phase before the official kickoff so it can be tracked throughout the project execution. It’s also important to proactively manage the schedule to ensure resource capacity aligns to any schedule adjustments.

Scope in agile

Work delivered vs work planned

In an agile model, the team operates using an iterative approach with each iteration delivering an incremental product. The work delivered is directly associated with the backlog items assigned to the given sprint. However, this also serves as a metric that teams can use to better understand their planning and ability to target scope and deliver each sprint. This allows the team to plan more accurately in future iterations.

Productivity trajectory

In an agile model, planning can improve for each iteration as the team better understands the impact of known constraints and how to manage potential blockers. This leads to another useful metric, productivity trajectory, which is a measure of whether the team is getting more done over time relative to the scope of work in each iteration.

There are two ways to look at this KPI: project team vs. organizational team

• A project team consists of individuals from different organizational teams working together for the duration of the project lifecycle. The KPI indicates how well the team is working together (or not) over the life of the project with a goal to demonstrate improvement.
• An organizational team consists of individuals who are “homed” within the same organizational unit within a company (application development, product development, DevOps, finance) and who work on multiple projects with individuals from other teams. The KPI measures the overall performance of the organizational team on meeting its responsibilities across all its projects.

Capabilities or features delivered

Scope for an iteration is typically chosen so that a specific feature(s) or set of capabilities is delivered at the end. By tracking the business-defined features that are accepted and delivered for each iteration, you can understand your progress toward the end goal.

Planned capacity vs realized capacity vs scope delivered

In agile, the project is typically handled by a dedicated team that remains the same at every sprint. However, team members are able to reflect on their prior commitments and their ability to honor them. Did they deliver by their committed date or with time to spare? Did they take on too much work? Did their performance impact the overall performance of the project? It’s also important to note that depending on organizational structure, teams are often a composition of fractionally allocated instead of truly dedicated members, which makes understanding a team member’s commitments and actual capacity even more important.

Budget in Waterfall

Cost variance

Cost variance, a popular measure when discussing earned value management (EVM), serves to measure how a project is aligning to its spend plan throughout the duration of the project. With knowledge of where the project is in its lifecycle, cost variance can be used as a performance indicator. Measuring cost variance allows us to understand how far along we are in project spending and if we're trending over or under budget, which helps determine if any adjustments are needed.

Budget in agile

Agile team cost

This refers to the cost of the agile team as a unit over time. (agile typically consists of a dedicated team, so team size should remain constant). With a constant team size, this measure can be used to determine the cost of work overtime by the development team if the amount of time needed to complete the work can be estimated. As part of the agile team cost, it can be worthwhile to understand the breakdown of resources by function (programmers, engineers, architects).

Requirements cost

With an understanding of the agile team’s structure, recurring costs and the scope of work, it's possible to break down cost estimates by specific features, epics, user stories or general capabilities as long as you have an estimate of the level of effort. By tracking this information, you're able to track not only the total product estimate but also the MVP estimate.

Some cost measures that also provide insight into value delivered include:

• Cost per delivered feature or capability = total cost to date / total delivered features or capabilities
• Cost per story point = cost to date / total delivered story points

An organization will often provide project funding for an agile project based on the amount of dollars estimated to deliver a set number of features. It's important to track the actual feature cost once delivered and compare it to the originally estimated cost.

Cost burn rate

This is the tracking of costs over a set period (sprint, month, release).

Schedule in Waterfall

Schedule variance (SV)

Measured similarly to cost variance, schedule variance indicates how ahead or behind a project is progressing against its planned schedule. By measuring schedule variance during a project, the team will identify schedule issues or risks and determine whether schedule compression techniques should be implemented (fast-tracking or crashing).

Schedule in agile

Velocity

Velocity for a Scrum team is a key planning tool and describes the quantity of work a team can be expected to deliver at the end of a sprint. From this velocity, the product owner conducts release planning to predict when a feature or multiple features can be delivered. This is often measured through a burndown or burnup chart at the sprint level.

Velocity predictability

Velocity predictability is measured as the difference between planned and completed velocity, which is the difference between the total planned and completed story points for a given sprint. By analyzing the fluctuations in velocity for a given sprint, one can understand whether persistent interruptions (competing priorities, blockers, unplanned work) are slowing down productivity.

Quality in Waterfall

Defect distribution

Defect distribution is key to measuring and improving product quality, and decreasing potential bug-fixing costs later in the project. In a traditional model, all development will typically be completed before the quality assurance (QA) phase. This should capture any major bugs before user acceptance testing (UAT). Any post-production defect statistics will serve as a measure for the QA/UAT and the development quality of the project.

Defect distribution can also be used to capture the test effectiveness, which highlights the difference between the number of defects found by development and QA and the number of defects found post launch.

Defect data can be distributed across multiple charts and tools for further analysis (Pareto charts, histograms, pie charts) and can be used to pinpoint areas that aren’t meeting your project’s quality standards.

Customer satisfaction

Another measure of a project’s quality is customer satisfaction. This can be quantitatively measured depending on the project’s end result through metrics such as a net promoter score and user churn rate if your product is a service. On a more qualitative note, customer satisfaction can be measured by what the project solves. Is it improving or replacing an existing process? Does it reduce costs? Has it received positive customer reviews? Customer satisfaction doesn’t only pertain to the end user. In a traditional project, it's important to keep the sponsor and key stakeholders involved to understand if the project achieved its original objectives. Defining the objectives and KPIs upfront allows the PM to measure throughout the project and make adjustments to ensure the objectives are achieved when the project is completed.

Quality in agile

Defect distribution and test coverage

In agile, testing and development occur in the same iteration. This allows the team to identify and fix bugs continuously, which lowers defect rates during the project. Tracking and eliminating defects over time will lead to a more production-ready final product. Having insight into the amount of testing being conducted and correlating it with the value delivered will improve overall project quality.

These are some variations to consider when measuring test coverage:

• Percentage of tests that are automated
• Number of automated tests per capability, feature, epic and/or story
• Percentage test coverage of capabilities, features, epics and/or stories
• Percentage test coverage of code (by module or line)

First-time pass rate

The first-time pass rate is a measure of the number of stories, features or capabilities that pass after the first test. Tracking this pass rate provides insight into how well user requirements are captured, understood, developed, integrated and delivered. It’s a good measure of the effectiveness of the end-to-end delivery process.

Summary

Conclusion

These KPIs are essential metrics to measure the effectiveness of your delivery. Although many of these KPIs are relevant for both frameworks, it's clear that the benefits of these KPIs and how to measure them can range depending on whether you follow an agile or traditional model. In a traditional project, where the focus is on a one-time final delivery, these KPIs reiterate the importance of planning and measuring effectiveness during the project. With a more flexible and iterative framework like agile, it's possible to use these KPIs for gradual improvement over each iteration and improve your overall delivery.

— By Victor Benavides