As quality engineers, we’ve spent years testing software with a simple mental model:

Given input X, expect output Y.

It works because traditional software is predictable. We know what goes in, we know what should come out, and we can write an assertion to verify it.

AI changes that in two ways:

1. The input space is almost limitless. Users can ask anything, phrase it in countless ways, switch languages, change context, or behave unpredictably.
2. The output isn’t deterministic. The same input can produce different responses across runs, and there may not be a single “correct” answer to compare against.

So the traditional question, “Did the system do exactly what we expected?”, is no longer enough. For AI products we need to ask:

“Did the system behave well within the accepted parameters, according to what we actually care about?”

That’s where AI evals come in, and it’s an area where quality engineers already have many of the skills needed to make a real impact.

What are AI evals?

AI evaluations, or evals, are a systematic way of measuring how well an AI system behaves against defined expectations. Instead of checking whether a response exactly matches an expected one, we check whether it meets the criteria we care about, such as:

• Correctness and completeness
• Relevance and tone
• Safety and policy compliance
• Correct tool usage
• Hallucination

And this brings us to an important distinction i.e. Model evals vs Product evals

Model evals vs Product evals

Model evals tell us how capable a model is in general: reasoning, coding, answering knowledge questions. They’re useful for comparing models.

Product evals tell us how well the AI works in our product.

Suppose I’m building an AI customer-support assistant. A model might score extremely well on general benchmarks, but that doesn’t tell me whether it can handle:

“My refund has been pending for 15 days. What should I do?”

For my product, I’d expect the AI to:$

• Check the refund status
• Follow the company’s refund policy
• Recognize that the SLA has been exceeded
• Escalate the case if required
• Avoid promising something it can’t guarantee

That’s a product evaluation. The question shifts from “How good is the model?” to “How well does the AI work in our product?” And this is where QE’s can contribute the most.

So, how would a quality engineer evaluate an AI product?

I’ll use the customer-support assistant as the running example.

The first thing I wouldn’t do is sit down and write 1,000 test cases. I’d bring the team together:

AI evaluation isn’t a QE-only activity. It’s a team sport.

Step 1: Build a reference dataset

Start with a small set of realistic scenarios and the behavior you expect from each:

This is your reference dataset. Importantly, it shouldn’t contain only happy paths. Include:

• Normal cases
• Edge and negative cases
• Ambiguous inputs
• High-risk scenarios
• Previous production failures

Start small and grow it as you discover new failure modes.

Step 2: Define a rubric

Now comes a more important question: how do we decide whether an AI response is good?

Imagine two quality engineers review the same response. One says, “This looks fine.” The other says, “No, it missed an important requirement.” Neither is necessarily wrong. We just never defined what “good” means.

A rubric is a set of clear, measurable criteria that tells us how a response should be evaluated. Start with the objective:

Evaluate whether the AI provides a correct, policy-compliant response and takes the appropriate next action for a refund request.

From that, we get our criteria: correctness, policy compliance, completeness, escalation, and hallucination. Here’s how three of them might be scored for the “pending 15 days” scenario:

Each scenario in the dataset gets tagged with the criteria that matter for it. The refund-API-down case is judged mainly on hallucination and correctness, while the 15-day case is judged on correctness, policy, and escalation.

This is why a single pass/fail isn’t enough. An AI can give the correct refund status and still fail because it didn’t escalate an overdue refund, made up a delivery date, or ignored the company’s policy.

Step 3: Run it, and learn from the failures

Run the assistant on every scenario, log the outputs, and have a domain expert review them. Then look across the failures for patterns. Several failures that look different often share one root cause, such as “no escalation logic for overdue refunds.” The patterns tell you which criteria deserve an ongoing metric and which are one-time fixes.

A reference dataset with a rubric is a solid start, but it doesn’t scale:

• Every change to the prompt, model, or retrieval means re-reviewing outputs.
• The dataset is finite, while real user input is not.
• Outputs vary run to run, so a scenario that passes today can fail tomorrow.

So the same rubric needs to be applied in more than one way. In practice we combine three approaches, each with a different purpose.

Human evaluation

Humans are usually where we start. Domain experts and product managers review responses against our criteria, and their judgment establishes what good looks like, what counts as failure, and whether the rubric even makes sense. These reviewed examples become the reference for automated evaluation later.

‍
Code-based evals: start with what you can measure

For each criterion, the first question should be: can I check this with deterministic code? If yes, that’s usually the best place to start. For example:

• Was the correct API or tool called?
• Is the response in the expected format?
• Is a required field present?
• Does the response contain information it shouldn’t?
• Did a specific business rule pass?

These checks are fast, cheap, and consistent. If code can reliably decide whether something is correct, there’s little reason to use an LLM for it.

LLM-as-a-judge: when judgment is required

We can easily check whether the refund API was called. But was the response relevant, complete, or appropriately worded? These are semantic qualities that simple assertions can’t capture.

‍

‍

‍