All Collections
Setup
Advanced Setup
Using Mean vs 85th Percentile
Using Mean vs 85th Percentile
Julian Colina avatar
Written by Julian Colina
Updated over a week ago

In this article, we'll go over different ways of calculating Change Lead Time and go over their pros and cons.

There are 2 types of distribution functions. One is symmetrical, other is long tail distribution (there are more types but out of context).

Symmetrical Distribution

Symmetrical Distribution

Long Tail Distribution

Long Tail Distribution

We want to see more signals less noise. Averages are useful when you don't expect outliers and if we show average in a long tail distribution, we'd not capture the long tail (outliers). Any improvements on the long tail AKA predictability will also not be visible.

The following metrics provided in Haystack by nature has a Long Tail Distribution. Due to this Haystack uses 85th percentile to calculate these metrics.

  1. Change Lead Time

  2. Development Time

  3. Review Time

  4. Release time

  5. Full Resolution Time for Bugs

  6. Time to Respond

Why 85th percentile?

We'll be focusing on Change Lead Time metric and its distribution, but the base logic can be applied to all metrics mentioned above.

Change Lead Time is time from first commit to merging the pull request. Due to the nature of how developers work, we can see Change Lead Time being a long tail distribution.

Cycle Time Distribution

Change Lead Time Distribution

Average (mean): 1.62 days

Median: 0.91 days

85th Percentile: 3.26 days

If we use the average Change Lead Time of the pull request above, we will get 1.62 days Change Lead Time.

Average is a good indicator of understanding the total time spent on the pull request and if the team is getting better using trends.

However, if we want to answer the question "How long would a new pull request take?", it will fall short of answering this question. Average is skewed by lots of short a few minutes long pull requests.

If we look at median, we can say with 50% confidence that this pull request will take 0.91 days. Using the median is less practical since we will be ignoring all the longer pull requests.

To answer the question, "how long would a new pull request take?" we should use 85th percentile. Essentially this means that with 85% confidence, we can say that the given pull request will be merged within 3.26 days. On the flip side, it also means that out of 100 pull requests about 15 will take more than 3.26 days to complete.

Using 85th percentile gives us

  1. Practically point of view on how long a feature request takes

  2. The result doesn't get skewed by extreme outliers - like a pull request merged 3 years later. (Note: if using mean, this will create a spike in the graph)

  3. The result doesn't get skewed by lots of shorter pull requests which most likely have lower business value. 

⚠️ Change Lead Time consists of Development, Review and Release Time. Because each metric is calculated by 85th percentile individually, sum of Development, Review and Release time won't be equal to Change Lead Time.

                       85(A) != 85(B) + 85(C)

Conclusion

At Haystack, we want to give more actionable insights and decrease the amount of noise there is in the data. Looking at 85th percentile for Change Lead Time is an industry-standard among Product Managers. We have analyzed millions of pull requests and found that 85th percentile gives more signals and less noise.

We recommend using 85th percentile when calculating Change Lead Time allowing you to understand and take action on your software delivery pipeline.

If you require further assistance, contact [email protected] or chat with us through the chatbot. We have helped 100+ engineering leaders track these data, and would be happy to share our expertise.

Did this answer your question?