From Days to Minutes: System-Level QA for Chatbots at Scale

This article describes how to test and evaluate a chatbot in minutes instead of days. The same coverage can be rerun after every change, with data showing whether the system is getting better or worse.

Executive summary

This system exists to make release decisions fast, repeatable, and evidence based decisions. Without it, chatbot quality, latency, and reliability regress silently as systems grow.

In our case, a single regression previously required days of manual effort across a wide and complex system. Today, the same coverage runs in under an hour and spans every major business domain.

This removes the release bottleneck, allowing teams to decide whether to ship or stop within hours using data instead of intuition.

At scale, this is not a testing improvement. It is operational risk control.

The problem we are trying to solve

Testing chatbots does not fail because teams lack ideas. It fails because of scale.

There are two scaling problems. The first is system scale. A chatbot is not a model, it is an end to end system with users, sessions, tools, services, and dependencies. The second is question scale. Testers need to ask hundreds or thousands of questions repeatedly, across users, tiers, and domains.

Manual testing breaks immediately. It is slow, inconsistent, and impossible to compare over time. It also cannot answer basic questions about latency, error behaviour, or system health.

What breaks without this

Without a system like this, failures do not announce themselves.

Quality regressions reach production because the same questions are not rerun consistently. Latency degrades gradually and is only noticed after user complaints. Errors caused by tools or dependencies are misattributed or ignored. The cost is not abstract. It is paid in time and manpower. Testers spend hours or days manually repeating conversations and comparing results by memory or notes. Developers ship changes without knowing whether they improved or harmed the system. Managers make decisions based on partial signals, delayed reports, or anecdotes.

As scale grows, coverage is reduced to cope with time constraints. This creates blind spots exactly where risk is highest.

The result is slower releases, higher operational risk, and decisions made without evidence.

What is actually being tested

This is system level end to end QA, not prompt testing. Each execution is a full system conversation that runs through authentication, session handling, orchestration, tools, and downstream services.

The goals are explicit.

Measure response quality. Measure errors and failure modes. Measure latency and execution time. Compare behaviour across versions and configurations.

If these signals are not repeatable, they are not test results.

End to end synthetic conversations

Instead of testers manually asking the same questions again and again, the system executes synthetic conversations. These are predefined questions executed in order and associated with specific users.

This addresses three distinct problems.

Testers stop repeating the same checks manually. Developers can see if their changes actually improved the system. Managers can see how the system is performing, not just anecdotes.

Question management at scale

Before this approach, teams had to carefully choose which questions were worth asking, because every question increased manual effort and execution time. With structured question management and automation, that constraint disappears. Teams can add, edit, and organise questions freely and confidently, knowing the system can handle scale.

Questions are managed as first class test assets. They are flexible and fully editable by testers or developers. Each question has the following properties.

Associated with a specific user, because different users may receive different information based on tier or permissions. Unique per user and conversation, meaning the same user cannot ask the same question twice in a single conversation. Active or inactive, allowing safe inclusion or exclusion without deletion. Versioned, so changes are explicit and traceable. Owned by a tester, for accountability and maintenance. Associated with a business domain, used for filtering, execution, and reporting. Associated with exactly one outcome per execution, either a response or an error. Can include golden standards, which represent desired responses. Can include a checklist used by a judge to score the response. Can include additional directives passed only to the judge for scoring purposes.

The base prompt is global to the system. Per question behaviour is controlled only through additional directives.

Runs and execution model

A run represents one execution of the system under test. A run consists of a selected list of questions.

Questions can be filtered by user, business domain, and owner. Each run defines the judgment mode, such as golden standards, checklist, or both. Each run captures a snapshot of the entire system, including service versions and configuration.

Questions are executed as conversations, in order, per user. Concurrency is handled by the runner and is not part of the data model.

Cost, load, and execution time control

To protect the system and control cost, the number of concurrent runs is limited. Each user requires an active chat session. The system ensures that a user session is not shared across runs at the same time, preventing clashes and corrupted conversations.

Results

Every question execution produces a result record. Each result has a unique identifier and is immutable.

Judgment in this flow is fully automated by the judge system, referred to here as the judgment mechanism. The judge system is triggered as part of the run execution, using the configured criteria, and produces repeatable scores without manual intervention. The same judge system can also be triggered independently outside this flow.

A simplified view of the result includes: user, question, response or error, score, justification.

Each result is linked to its run and to the system snapshot used during execution.

System events and observability

For every question, the system collects relevant events from across the flow. This includes information from tools and services involved in answering the question. These events provide the context required to explain errors, latency, and unexpected behaviour.

Domain-based reporting

Results are aggregated by business domain. This allows identification of domains that are problematic, even when conversations span multiple domains. Domain attribution follows predefined logic so reporting remains consistent and trustworthy.

Run statistics

Each run produces statistics at two levels, per business domain and for the entire run.

Key metrics include: failure rate, latency, error types, score distributions.

These metrics describe both quality and system health.

Release gating and decision rules

Measurement alone is not enough. At scale, results must drive clear go or no go decisions.

This system does not hard code release rules. Instead, it provides direction for defining generic, transparent gates that leadership and engineering agree on upfront and do not change ad hoc during a release cycle.

Typical release gates include.

No new error types compared to the previous baseline run. No statistically meaningful regression in p95 or p99 latency. No drop in judge scores for critical business domains. No increase in failure rate beyond an agreed tolerance.

These gates turn test results into decisions. If a gate is breached, the release is blocked. If all gates pass, the system is considered safe to ship.

Because gates operate on run comparisons, decisions are based on trends and deltas, not isolated results.

Comparing runs and measuring progress

Runs are comparable over time. Run A can be compared with Run B using the same questions and judgment mode. This makes it possible to track progress and regressions.

Comparisons consider: response quality, error presence and types, latency changes.

Because each run is tied to a system snapshot, changes can be correlated with specific version or configuration differences.

Who this solves problems for

This approach removes different pains for different roles.

Testers reduce regression cycles from hours or days to minutes by running automated, repeatable conversations instead of manual checks. Developers get objective answers to whether a change made things better or worse, without relying on gut feel. Managers get an aggregated, comparable view of system health across domains, versions, and time, instead of scattered reports and opinions.

What this enables

This approach provides an end to end QA solution for chatbots. It operates entirely at the backend level, without a user interface, to maximise stability, speed, and execution time. The chatbot is exercised exactly as a user would use it, but without UI flakiness or manual effort.

Results are aggregated and exported in formats suitable for analysis and reporting, such as structured data and spreadsheets. This enables clear summaries for managers alongside detailed drill downs for engineers.

It turns chatbot testing into continuous system level regression testing. It replaces subjective judgement with repeatable signals. Most importantly, it reduces decision time by making system progress, regressions, and risk immediately visible.

This article also appears on Medium