In this article, we'll go over different ways of calculating Cycle Time and go over their pros and cons.
There are 2 types of distribution. One is symmetrical, other is long tail distribution (there are more types but out of context).
We want to see more signals less noise. Averages are useful when you don't expect outliers and if we show average in a long tail distribution, we'd not capture the long tail (outliers). Any improvements on the long tail AKA predictability will also not be visible.
Long Tail Distribution
There are Haystack metrics that fit into both of these distributions. We'll be focusing on specifically Cycle Time metric and its distribution.
Cycle Time is from first commit to merging the pull request. Due to the nature of how developers work, we can see Cycle Time being a long tail distribution.
Cycle Time Distribution
Average (mean): 1.62 days
Median: 0.91 days
85th Percentile: 3.26 days
If we use the average Cycle Time of the pull request above, we will get 1.62 days Cycle Time.
Average is a good indicator of understanding the total time spent on the pull request and if the team is getting better using trends.
However, if we want to answer the question "How long would a new pull request take?", it will fall short of answering this question. Average is skewed by lots of short a few minutes long pull requests.
If we look at median, we can say with 50% confidence that this pull request will take 0.91 days. Using the median is less practical since we will be ignoring all the longer pull requests.
To answer the question, "how long would a new pull request take?" we should use 85th percentile. Essentially this means that with 85% confidence, we can say that the given pull request will be merged within 3.26 days. On the flip side, it also means that out of 100 pull requests about 15 will take more than 3.26 days to complete.
Using 85th percentile gives us
Practically point of view on how long a feature request takes
The result doesn't get skewed by extreme outliers - like a pull request merged 3 years later. (Note: if using mean, this will create a spike in the graph)
The result doesn't get skewed by lots of shorter pull requests which most likely have lower business value.
At Haystack, we want to give more actionable insights and decrease the amount of noise there is in the data. Looking at 85th percentile for Cycle Time is an industry-standard among Product Managers. We have analyzed millions of pull requests and found that 85th percentile gives more signals and less noise.
We recommend using 85th percentile when calculating Cycle Time allowing you to understand and take action on your software delivery pipeline.
If you require further assistance, contact [email protected] or chat with us through the chatbot. We have helped 100+ engineering leaders track these data, and would be happy to share our expertise.