Skip to main content

Quality Assurance Metrics

The goals of quality assurance and the quality assurance process are different for every company. You might want to search for many different behaviors, issues, and opportunities in conversations depending on your current business goals.

This article focuses on the high-level metrics that are relevant for any quality process in any company. You want to watch those no matter the specific use case to ensure you are using your time and resources efficiently.

Feedback Loop Duration​

The time it takes from a moment when something actionable happens in a conversation until it can be acted on. For example:

  • An agent is providing inaccurate information to the customers. How long does it take from the moment when the agent first provides that information until the agent is made aware that the information they provide is not accurate and starts providing up-to-date information? This example can be used for any behavior that requires talking to an agent.
  • The process causes customer frustration because an agent cannot help the customers. How long it takes from the first customers who are frustrated by the process until that process is fixed or at least the company is aware that it needs to be fixed.

The short feedback loop is important for these reasons:

  • Minimize blast radius. A shorter time to resolve the issue reduces the number of conversations affected by it. Whether the cause for the issue is an individual agent a process or a structural issue.
  • Agents respond better to timely feedback. When agents still remember the specific conversation. Any feedback feels less abstract and they can easily connect it to real-life situations.

Salted CX has data loads in 15-minute time intervals which enable you to do reviews during the day on conversations that happened a few minutes before the review. All review results are visible in dashboards after the next 15-minute load interval. Use these features to focus on the latest conversations and give everybody else in the company feedback during the day.

Review Coverage​

The percentage of engagements is reviewed by a person. It is important to balance the review coverage with invested effort. For detailed reviews, the industry standard is commonly around 1% of conversations that get a review. This percentage may be much higher for a specific subset of conversations — for example when ensuring compliance is critical in those conversations.

Higher review coverage means that you have a higher chance of uncovering a situation you need to act on. Review coverage is simple to calculate but needs to be treated carefully as it might be expensive and inefficient and has diminishing results unless targeted on specific conversations.

There are several ways to increase the review coverage:

  • Increase the number of people doing reviews. This is an obvious method to increase the review coverage. We mention it for completeness. Typically you should focus on doing more with fewer people to provide good value for money.
  • Involve people from other departments in reviews. This is similar to the previous one with the exception that there might be synergies where people use their time to help both departments. Are other stakeholders interested in how customers, talk about their product? Is the customer experience department interested in why some customer journeys do not end well? Does your AI/ML team need annotated data? Let them collaborate on customer journeys and provide their reviews and tags. These can help you better understand what other departments expect from the customer journeys and decide what to focus on next.
  • Read conversations chronologically as they happen. Often legacy quality assurance is driven by the need to answer questions in a form. This leads to revising
  • Provide only actionable feedback. In legacy quality assurance you often have to respond to many questions in a form in case nothing exceptional happens. If agents are trained appropriately the number of such cases should be a vast majority.
  • Use short and very focused forms. Based on the selection criteria for the conversations above use forms that are designed for the issues that you are most likely to encounter and do not contain unrelated, rarely encountered tags and questions. This makes it easier to navigate the form.
note

The goal should rarely be to review 100% of conversations unless it is a very specific subset of conversations that are critical to review. The ideal aspiration scenario is to review 100% only of conversations that are “worth reviewing” — they contain something that you can act upon. However, this is difficult to measure in most cases.

Actionable Findings per Effort​

The number of actionable things you have uncovered for the effort you have invested. Ideally, you would be also able to express the value of the findings but that might not always be straightforward or possible to do accurately. Effort is typically expressed by the time necessary for doing the manual reviews.

You can recognize actionable findings very easily by the fact that you know where to go and what to do about it — such as going to an agent and telling them to change their behavior, going to the product department and telling them what issues customers recently encountered with their product, etc.

The ways to increase actionable findings per effort:

  • Focus on conversations that have a higher probability of having actionable issues or opportunities based on metadata. For example conversations with excessive hold time, poor customer satisfaction, findings by AI in conversation content, etc.
  • Use short and very focused forms. Based on the selection criteria for the conversations above use forms that are designed for the issues that you are most likely to encounter and do not contain unrelated, rarely encountered tags and questions. This makes it easier to navigate the form.

Alignment with Customer Expectations​

In most cases, you want to have at least some quality aspects aligned with the customer’s expectations. So if customers consistently express low customer satisfaction with an agent but your quality process shows that the agent is doing great in customer experience-related questions you need to check the quality process.

Issues with misalignment between customer satisfaction and quality:

  • On individual engagement level. The customer provided feedback after an engagement. At the same time, the engagement was reviewed by a person and was reviewed with a different outcome. The best next step is to look into the customer journey to decide whether the quality assurance process worked as expected or not.
  • On average for agent, team, queue, or other aggregated level. If you provide quality reviews consistently with customer satisfaction on individual engagement level but you see the difference when looking for an example on agent performance the cause is likely in the sample that you are selecting. The sample may not be large enough or it may not be random — which may be intentional if you are focusing on conversations that are potentially problematic.

Note that customer satisfaction may be influenced by many factors, some are out of your control and there are some you might not be aware of. It would be very rare to be able to understand and address every single misalignment between your customers and your perspective. Always consider the scale on which the misalignment happens for the decision of whether to focus on it.

note

Customer satisfaction might not be aligned with all the quality questions you try to answer. Customers for example might not be necessarily concerned or aware of the regulations you have to follow. So make sure you compare those quality questions that should be related to the customer experience.

Do Not Review Unnecessary​

For every case you use manual reviews for you should always consider these two options:

  • Another data source already contains the information. In some instances, you might be reviewing manual reviews to find information that is already available. Or even when it is not available for every conversation it might be available for a large enough sample that manual reviews are unnecessary or their scope can be much smaller.
  • AI can do the job with acceptable accuracy. Depending on the use case you can use AI to review 100% of conversations. You need to provide feedback in case the AI is inaccurate which significantly minimizes scope.