Change Failure Rate tells us the percentage of deployments that created a production failure - this tells us how robust and reliable our deployment process is.

In this article we'll be looking into:

1. How to Configure Hotfix Deployments (Change Failure Rate)
2. Common Questions

Step 1: Go to Settings

Click on the 'Settings' tab in the left sidebar to open up Haystack settings.
​

Click on 'Configure Metrics' in the left sidebar. And then click the 'View configuration' button under the 'Change Failure Rate' card.

Click on 'Hotfix Source' to select the source to track Hotfix Deployment. You can select either Jira or Version Control as a hotfix source.

If the version control option is selected you will see the list of repositories. You can configure change failure rates by a specific repository or all of them at once by clicking the `Configure All` button.

After clicking configure button next to the repository you will be redirected suggestions section to select any predefined configuration that fits your organization. You can either select any predefined suggestion or click the custom config option to select your own method.

After selecting custom configuration, click on the first dropdown. There are 5 options you can choose to calculate Change Failure Rate.
​

Option 1: containing title
Any pull request where the pull request title contains set as X will be counted as a single single hotfix.

ex: "hotfix", "rollback", "revert" etc.
​
Most common terms used by engineers are "Hotfix", "hotfix", "Fix", "fix", "Rollback", "rollback", "Revert", "revert", "Bug", "bug".
​
​Note that Haystack does exact string matching.

Option 2: containing label

Any pull request which has label X will be counted as a single hotfix.

ex: "release", "rollback", "revert" etc.

Option 3: containing branch

Any pull request with head branch X will be counted as a single hotfix.

ex: "release", "rollback", "revert" etc.

​Option 4: each commit containing git tag

Any commit tag containing tag X will be counted as a single hotfix.

ex: "release", "rollback", "revert" etc.

Option 5: each tagged commit not containing git tag

Every tagged commit that doesn't contain tag X will be counted as a single hotfix.

ex: "prod-", "staging-" etc.

If the selected method is Jira you will see the configuration page below. You can narrow down the issues by adding filters like issue type, priority, or any custom field you wanted.

After every change, the preview table will reflect the related issues with the given configuration.

Note: If Jira is selected as hotfix source, that means the configured rule will be applied to all the repositories.

Step 5: That's it!

The changes might take a few minutes to calculate. Go back to deployments page scroll down and see your organization's Change Failure Rate!

Yes, you can track each repositories' Change Failure Rate using different rules. All you need to do is click on the `Configure` button next to each repository.

Haystack supports multiple terms only for Change Failure Rate. Note that you'll need to press enter after each term.
​
​What are the best practices for tracking Change Failure Rate?

The goal of tracking Change Failure Rate is to see the quality of the releases. The ideal calculation would be either through CI or manually tagging each hotfix Pull Request.
​
If your team has less than 50 releases per 6 months, we recommend manually adding labels, and creating a process for tagging hotfix releases.
​
If your processes are more frequent and more complex, we recommend using proxy calculation methods to see the trend of Release Quality. This can be achieved by checking terms in pull request titles like "bug". The calculation doesn't necessarily need to be 100% accurate. As long as you can find common terms that your engineers are using, it'll give you an idea of the Release Quality, and spur conversations about it.
​
If you require further assistance, contact [email protected] or chat to us through the chatbot. We have helped 100+ engineering leaders track these data, and would be happy to share our expertise.

Check the following blogs: