As an engineering team, having great process and technology helps you deliver value to your business quicker. Businesses have metrics like revenue to track how well they perform, but how can you do the same for your engineering team?
Engineering teams often risk measuring the wrong things. Focusing on overly local metrics (like, say, lines of code changed per Pull Request) could lead to you making changes that degrade overall engineering team performance at the expense of a meaningless local optimisation.
First and foremost, Haystack lets you track "North Star" metrics to get a global picture of how your engineering team is performing. They track the entire picture of your engineering team, whilst still being metrics that are in the control of engineering leaders. These metrics have been demonstrated to reflect an engineering team's ability to deliver business value. Indeed, if you've read books like "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" you may already be familiar with the power of these North Star metrics.
Getting the right measurements is one thing, but you also need to be able to drive improvements to these metrics. Haystack allows you to measure the "Cycle Time Metrics" that form the "Cycle Time" North Star metric to identify if slowdowns are occurring in the development process or the code review process. This review time can then be further subdivided into "Review Time" metrics to help diagnose where slowdowns are occurring in the code review process (formed of First Response Time, Rework Time & Idle Completion Time).
To prevent burnout, Haystack also provides "Throughput" data, to identify to help identify a healthy pace for your engineering team and preventing developers from getting burnt out.
"Leading Indicators" can help give an early indication of trends that could impact your North Star objectives. These early indicators are vital both in helping drive improvements quickly, and at stopping poor practice from becoming a systematic issue. Haystack allows you to prevent Technical Debt before it happens by alerting on these risk factors.
North Star Metrics
North Star metrics give the team high-level goals on which to align. These north star metrics act as beacons - highlighting potential problems as they occur.
Cycle Time is one of the most powerful and holistic North Star metrics. Cycle time is the time from first commit to pull request merged.
Cycle Time measures the critical parts of the product delivery process that are directly under the control of the engineering team. Accordingly, constraints like technical debt, team burnout, context switching and poor software engineering process are ultimately reflected in this metric.
Cycle Time lets us understand an engineering team's speed and tempo. It gives us a picture into how quickly work a team is able to complete work, teams with shorter cycles are more able to experiment quicker and be more agile to changing business needs. Teams should aim for shorter cycle times.
Definition: Time from First Commit to Pull Request Merged.
Average: The average Haystack team has a Cycle Time of less than 3 days.
Haystack allows you to dive much further to find bottlenecks. In the Haystack Dashboard, you can drill down to understanding where time is spent during the engineering process, including:
Deployment Frequency helps identify the rate at which you are delivering new business value to your customers. Smaller deployments have less risk of going wrong and provide an opportunity to deliver value to your customers in shorter iterations, allowing you to learn quicker.
Definition: What percentage of deployments require a hotfix.
Average: Most teams deploy at least once a day.
Change Failure Rate (CFR)
Change Failure Rate tells us the percentage of deployments that created a production failure - this tells us how robust and reliable our deployment process is.
Definition: Percentage of deployments which caused a failure in production.
Average: Typically less than 15%.
Throughput (Detecting Burnout)
Throughput gives us a sense of the team's bandwidth. It gives us a picture into how much work we can typically accomplish. Teams should aim for consistent throughput.
Definition: Number of merged pull requests
Average: The average Haystack team has a weekly Throughput of at least 3 pull requests per member
Haystack provides an overview as to what a team's Throughput is and dynamically calculates a Healthy Area. Team's which exceed the healthy range risk burnout. Haystack can even send you notifications when your team is at risk of burnout using the Weekly Recap notifications.
Below you can see an example of a team's throughput dropping below the Healthy Area in the week following a dramatic increase in throughput. This is a clear sign of team burnout.
Haystack additionally provides a set of drill-in metrics for Throughput:
Average Throughput: Average Pull Requests merged in a given timeframe
Merge Rate: Merged Pull Requests as a percentage of all open Pull Requests
Close Rate: Closed Pull Requests as a percentage of all open Pull Requests
Leading Indicators provides an insight into some rule-of-thumb metrics that we have found generally help drive North Star metrics. In other words, we know that improving these metrics helps drive improvements in North Star metrics. Conversely, when these metrics degrade, they provide a risk factor that your North Star metrics may soon worsen too. These leading indicators provide real-time, fast-turnaround insight into how to improve and maintain your North Star metrics.
Leading Indicators only form a small part of the way you manage risk as an Engineering Manager. Using Haystack's notifications feature, you can receive proactive daily health checks that will warn you on multiple other critical risk factors, from Pull Requests being merged without review to Pull Requests stuck in back-and-forth discussion.
Pull Request Size
Pull Request Size gives us a sense of how large or complex pull requests are. Teams should aim to work in small batches and maintain an average pull request size of less than 200 lines of code.
Definition: Number of lines changes per pull request
Average: The average Haystack team has an average pull request size of 200 lines of code.
Work In Progress
Work in Progress (WIP) gives us a sense of how many outstanding pull requests are actively being worked on across the team. Teams should aim to reduce WIP per member and maintain an average of no more than 2 pull requests per member.
Definition: Number of open pull requests
Average: The average Haystack team has an average WIP less than 2 per member.
Weekend Activity gives us a sense of how many members feel the need to work over weekends. This gives us a sense of how overworked the team is. Teams should aim to reduce weekend work.
Definition: Sum of github activities (commit, comment, push) over the weekend
Average: The average Haystack team has less than 1 weekend activity on average.