Measuring Your Team's Software Delivery Performance
I recently worked with an organization that was looking to make changes to their software delivery processes so they could deliver higher quality software more quickly. We used research and guidance from Google’s 2024 DORA Report to drive those changes.
The goal of the report is to identify ways that technology can deliver value to organizations and their customers. The content of the report was driven by yearly surveys completed by over 31,000 IT professionals worldwide.
One area of the report I found especially interesting was the survey questions related to the Four Key Metrics of software development.
##Throughput survey questions ###Deployment frequency For the primary application or service you work on, how often does your organization deploy code to production or release it to end users? ###Lead time for change For the primary application or service you work on, what is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)?
##Stability survey questions ###Time to restore service For the primary application or service you work on, how long does it generally take to restore service when a service incident or a defect that impacts users occurs (e.g., unplanned outage or service impairment)? ###Change failure rate For the primary application or service you work on, what percentage of changes to production or released to users result in degraded service (e.g., lead to service impairment or service outage) and subsequently require remediation (e.g., require a hotfix, rollback, fix forward, patch)?
The table below captures the results of the survey.
| Aspect of Software Delivery Performance | Elite | High | Medium | Low |
|---|---|---|---|---|
| Deployment frequency | On-demand (multiple deploys per day) | Between once per day and once per week | Between once per week and once per month | Between once per month and once every six months |
| Lead time for change | Less than one day | Between one day and one week | Between one week and one month | Between one month and six months |
| Time to restore service | Less than one hour | Less than one day | Less than one day | Between one week and one month |
| Change failure rate | 0-15% | 0-15% | 0-15% | 46-60% |
| *This table is provided courtesy of the ACCELERATE state of DevOps 2019 report |
Looking at the results, a couple things stand out to me:
- Elite performers deploy frequently.
- Elite performers release bugs to production just as frequently as High and Medium performers. The difference is they get them fixed a lot more quickly.
Comparing Elite performers to Low performers, the 2019 report finds:
- Elite performers have 20 times more frequent code deployments
- Elite performers lead time from commit to deploy is 106 times faster
- Elite performers recover from service incidents 2,604 times faster
- Elite performers are 1/7 as likely to deploy failing changes
##Wrap-up These metrics are a great way to get a baseline of your organization's software delivery performance, and a good way to quantify the success of your agile transformation activities.