Accepting a Business AI System: Tests to Agree Before You Build

Agree the evidence before the build.

Describe the unit of work, the user, the permitted inputs and the action or answer expected. Then define failure, including behaviour that is unacceptable even if the task otherwise succeeds. Put those rules in the acceptance record with the environment, dataset version, owners and required evidence. A demonstration chosen by the builder is useful for explanation; it is not a substitute for the buyer’s test set.

The people who do the work should write or select the cases and judge the business result. Source owners write citation and freshness cases; access owners write permission cases; operators write recovery cases. The builder translates them into repeatable checks and explains technical limits. Keep final sign-off with the named buyer-side owners.

Keep four kinds of cases.

• Real work. Use a permitted, representative sample of current cases, with the expected outcome and necessary context. Remove or protect sensitive details under the agreed data policy.
• Hard work. Include incomplete inputs, exceptions, conflicting documents, long records and downstream failures. Label the expected escalation where no single answer is justified.
• Should-refuse work. Ask for unsupported answers, out-of-scope tasks and actions beyond authority. Assess what the system exposes while refusing, not only the refusal sentence.
• Permission work. Run the same request as users with different rights, including someone whose access has just been revoked. Inspect retrieved passages, citations, tool effects and relevant logs.

Separate development cases from a held-out acceptance set. Record where expected answers came from, who reviewed them and which source version applied. Resolve disagreements in the rubric before grading the system. Add failures found during use to regression coverage while retaining a fresh held-out set for later decisions.

Use a scorecard for the system you are buying.

The sections below cover agents, knowledge systems and private infrastructure. A system spanning them needs the relevant checks from each. The pass-rule cells are deliberately blank: agree the rule, denominator, test conditions and critical failures for your own process. Print the tables or copy them into an acceptance record you control.

Separate release blockers from graded quality. A strong average answer score must not conceal a tested permission leak or an unauthorised action. Count those events explicitly, investigate them and define the stop rule before testing. A clean test set provides evidence within its coverage; it is not a guarantee about future inputs.

Compare in shadow mode before granting authority.

In shadow mode, the system proposes answers or actions without making production changes. Compare its proposal side by side with the current human process on the same permitted cases. Record corrections, review effort and missed exceptions as well as time to completion. Avoid showing one reviewer the other answer first if that would bias the judgement.

Shadow mode cannot exercise every production effect. Use a controlled environment for tool writes, approval outages and compensation drills. Document differences from production, such as synthetic identities or unavailable integrations. Decide which checks must be repeated under limited production access before expanding use.

Use today’s work as a baseline.

Score the current process with the same rubric, including disagreements and costly errors. Set the acceptance threshold from business consequences, review capacity and required controls. A high-volume low-impact draft task may tolerate corrections that an external commitment cannot. Do not copy a vendor’s published score into the contract without matching its task, dataset and error definition.

Specify the denominator: all submitted cases, only answerable cases, or all material statements. Record abstentions separately so the system cannot appear accurate simply by declining most work. Break results down by difficult cases and user groups. Small samples are useful diagnostics; state their limits and collect more evidence where the decision remains uncertain.

Own a regression suite you can rerun.

Retain the permitted test inputs, expected results, rubric, runner and result files in a location the buyer controls. Pin model, prompt, retrieval, index, policy and tool versions in each run. Rerun relevant checks after changes to any of those, including serving settings and source permissions. Review regressions before release rather than relying on a familiar model name.

Version the rubric and pass rules too. If an owner changes a rule, record why and rescore the baseline where feasible. Keep the previous accepted configuration and a tested return path. A model swap is a new evaluation event, even if the application API stays the same.

SYNTHETIC filled example.

This is an invented support-knowledge system for a fictional internal support desk. It answers staff questions from a bounded policy library; it sends no customer messages. The cases, counts, evidence-pack references and role sign-offs below are fabricated to show the method. They describe no client, project or measured Axis Bridge result.

SYNTHETIC decision: do not accept this version. The citation test failed despite the other passes. Investigate case 7, repair the cause, then rerun the citation test and the relevant regression suite before a fresh sign-off. This is an illustrative example: the evidence packs and role names are fictional, and nothing here is a signed record.

Sign a version and a scope.

Record the accepted configuration, dataset and rubric versions; results and evidence locations; exceptions and their owners; permitted users and actions; unresolved risks; operating owner; and rollback or compensation procedure. Include the names and dates of the authorised signers. A conditional acceptance needs a restriction and a condition for lifting it, rather than an unexplained waiver.

For a scoped implementation, see AI agent development, RAG and knowledge systems or private AI infrastructure. Bring the task examples and the acceptance record so the discussion starts with evidence your team needs to see.